The subject say it all but let me explain in a bit greater details.
In older version of IE brushtool use to work on mask layer (not tested in 9.0) but in 9.1 brushtool tends to loose its orientation when a user tries to apply brush strokes to selected layer if that layer is a Mask Layer.
Even if we convert a Mask Layer to plain layer by setting its IsMask := False and Visible := True the brushtool is still not working properly.
I think it has got something to do with ie8g. Just my guess... because if we set the PixelFormat of concerned layer to ie8p or ie24RGB the brushtool starts working as expected.