I have used a simple never-fail approach with my students, which I call the “n+sigma” rule. Here’s what you do:
draw the Lewis structure of the compound. Here is one possible structure for XeOF4:
2. look at the central atom (here, Xe) and classify all the electron pairs around it as either sigma, pi, or unshared (n). Remember that single bonds are sigma, double bonds consist of one sigma and one pi bond, and triple bonds consist of one sigma and two pi bonds. Here, sigma = 5, pi = 1 and n= 1.
Add sigma + n = 5+1 = 6. this is the number of hybrid orbitals that Xe will need to house these electron pairs; so here Xe needs 6 hybrid orbitals.
3. The central atom will use its s-orbital, its p-orbitals and as many of its d-orbitals as needed to mix together to make the hybrid orbitals. So, for XeOF4, Xe will need its s orbital, all three of its p-orbitals, and 2 of its d-orbitals, and its hybridization state will be sp3d2, or d2sp3.
There is another possible structure you can draw:
but this will still give you the same answer.
Try this approach for a number of different molecules; it works like a charm!