The recent development of IoT in many security-sensitive environments enhances the importance of security in all layers. Several IoT networks implement the Routing Protocol for Low Power and Lossy Networks (RPL) in the network layer due to its energy-efficient mechanisms. There are no basic security features; hence, RPL security has been focused on by IoT researchers. There are several routing layer attacks, such as blackhole and selective forwarding attacks, sinkhole attacks, Sybil attacks, wormhole attacks, DoS attacks, and acknowledgment spoofing. To solve such security issues, current security mechanisms are cryptography based solutions, Trust measurement, Game-theoretic approaches, and Intrusion Detection Systems (IDS). The cryptography methods employ public/private key pairs, authentication certificates, and pseudonyms to ensure network security. The trust-based model computes the trust of other nodes directly or indirectly for secure data transmission. The game-theoretic models consider the communicating nodes as game players and select the best player for routing. Finally, the IDS model monitors the node behaviors straightly or historically and detects the attackers successfully in the network. However, they become inadequate for effectively handling sophisticated security and intelligent attacks.