A common mistake worship leaders (myself included) make especially in the early years of worship leading is looking at the time we need to spend with God as a task rather than a relationship. There is a great example of this in Luke 10:38. Jesus visits Martha's home and while she's busy fussing about preparing a meal (focusing on task) Mary drops everything to sit at Jesus' feet and hang on every word he says. Martha get's frustrated at the fact that Mary is choosing to not help her and rest at His feet. As a result, Jesus basically tells Martha she's missed it, she views God's presence as a task rather than just enjoying it.
Who doesn't like to be served? If you likened your relationship to God as a marriage it might look something like this....Marriage is all about serving your other half but if all we ever did was bustle about serving each other and never sit to enjoy each other's company we would start to question the other persons motives for serving. God wants to enjoy our company, to speak to us just as much as we speak to him. He wants for us to go on a date with him rather than just give him a kiss on the way out the door.
Look at spending time with God as you would a best friend or spouse. It causes your feelings of inadequacy to fade and you will gain the confidence to lead from a standpoint of knowing God and his character.