I wouldn't assume that the device is going to operate gracefully when it can't draw enough current. You might put the radio portion of the module in an undefined state in that scenario. I've things like that happen with other Digi products.
Also, there are low power varients of the Xbee. they use 40mA, which would be well within your 100mA budget.
see pg.9 specs of low power unit here:
http://ftp1.digi.com/support/documentation/90000976_V.pdf