The solution for “touch: cannot touch `test.txt’: No space left on device”
In this guide, we are about to provide a simple solution for the titled issue.
In case we receive “No space left on device” our first assumption will be mount point may get filled. Yeah, I go through the same but while looking into the specific mount point we found it has enough space left free. By following our thought should target “File System error” even filesystem have no error we may move to check for open files if that is not the case, at last, we can check the Inodes. Yes, here we have used with 100% of inodes under /var. finally, we have located thousands of files under /var/spool/clientmqueue.
We are trying to touch a file under /var and get “No space left on device”.
[root@ServerProd1 var]# touch test.txt touch: cannot touch `test.txt': No space left on device [root@ServerProd1 var]#
While checking the space under /var we have enough space.
[root@ServerProd1 ~]# df -h /var/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/VG01-LogVol03 4.7G 2.0G 2.5G 44% /var [root@ServerProd1 ~]#
By following we have checked the file state. Even it looks good.
# tune2fs -l /dev/mapper/VG01-LogVol03 | grep 'state'
[root@ServerProd1 ~]# tune2fs -l /dev/mapper/VG01-LogVol03 | grep 'state' Filesystem state: clean [root@ServerProd1 ~]#
Check for the open files. Even we can check for the deleted open files.
# ulimit -n # find /proc/*/fd -ls | grep '(deleted)'
At last, we are about to check the inodes.
Yes, here we found 100% of inodes are utilized.
# df -hiP /var/
[root@ServerProd1 ~]# df -hiP /var/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VG01-LogVol03 313K 313K 0 100% /var [root@ServerProd1 ~]#
Now we have found thousands of files which small in size under /var/spool/clientmqueue once we clear them we are in good stand for inode utilization under /var. Those files are just mail queue and safe to remove in our setup.
[root@ServerProd1 clientmqueue]# ls -lthr | wc -l 309821 [root@ServerProd1 clientmqueue]# pwd /var/spool/clientmqueue
We have found the issue once we found the number of files consumed under /var/spool/clientmqueue. That’s it, here is the solution for “no space left on device” which caused by inode utilization.