I can see Marvolo Gaunt wanting to take a look at his collection. He works for Borgin & Burkes so may come there interested in buying or selling something his bosses want. He could even try to steal something if it catches his eye.

I also offer Edgar Minks who may be known as somebody willing to invest in the development of some inventions.. maybe somebody Xavier recommends for funding help? It's also possible he may just want a weird object.

If you really want to make him uneasy, Walter could come sniffing around, maybe got a tip Xavier keeps dangerous dark magic kind of stuff in his home. He won't have a warrant to search but he could come by and ask questions, maybe innocently ask about a tour.