I also think of it as a mix of 'Seishin is too polite for sneaking into someone's room unanounced' and 'well we're kinda fighting now so it couldn't be Seishin even if he's the most likely to visit me at this hour'

From how it looks like, I think even Toshio doesn't mean to fight with Seishin even if they just had a major disagreement, even if Toshio was the one to push Seishin away (at the moment of Moral Horizon). Seishin doesn't seem to me as the one intending to fight as well, he just left Toshio alone because Toshio told him to go back home. It seems to me that Toshio only begins to realize that Seishin's sort of 'left' when he knows that Seishin's surrendered himself to the Kanemasa, and only fully begin to consider the idea that Seishin's actually opposed him around the 'they're collaborators!' part I figure. From how it looks like he's being kinda slow, it rather shows his psyche which expects Seishin to always be there on his side, active or passive supporter. Even if there are occasions when he doesn't support Toshio, he expects Seishin to just stay on neutral ground, being a rather passive person he is. He's never imagined that they'll stand before one another as actual enemies before. That's how people are about their best friends I figure.