Hello KaitoWL!
Welcome to our fine community 
Because you said you already dove into tutorials and made some spells, you probably read through this: Getting Started in Real Time VFX? Start Here!
I always chuckle if I read something like “realistic spells”, because they are not real. And I think this is the hard part. Making something that does not exist, but let it look like it could exist in the real world.
As I do not know, what you are capable of now, I just assume that you know the base methods of creating spell effects (particle systems, textures, flipbooks, simple meshes, shader/materials, maybe basic scripting). Tutorials that have an “all in one package” are not very much and do cost a bit. But there are a bunch of good ones with specific topics, that you can find, if you know what to look out for.
The next step would be learning would probably:
-
learn to make more complex 3d objects.
Ice shards, stalagmites, floating rocks, hand of a summoned demon… all these things will need you to create more complex meshes and understand shaders/materials better. PBR workflow is crucial nowadays for “realistic”. -
learn to use simulations.
Smoke trails, liquid splashes or a rock breaking apart… you have to learn to make these kinds of simulations and import them into the game (through flipbooks, baked animation or VATs). -
have understanding on how real things work.
Reference is key. I bet most of the people here have a vault with a bunch of references of water, smoke, lava or whatever. Looking at those and dissecting what happens in the motion will give you a better eye for what the effect is missing to look more natural.
Other than that, I would recommend looking at some assets like the one’s from kripto289: kripto289 - Asset Store
These were good learning resources for me, as I looked at all the shaders, animations and settings that have gone into these effects. And they are mostly really well optimized with some great scripts to toggle even fps caps! (but do not use the newest version of unity. the creator updates it, but with each update of unity it breaks a bunch of the effects, because the scripts try to be useable for HDRP and URP as well).
Another Note: also search for tutorials in UE4. Even if the engine is different, the approaches and techniques are the same.