Youtube-dl called from python script fails...why ? [solved]

Question

Hi there,
since downloading youtube stuff with Browser plugins has been pretty unstable for me lately (read: mostly incomplete videos) I switched over to youtube-dl again. Unfortunately youtube-dl often stops downloading as well...
I wrote a small Python script which looks for incomplete downloads in my Youtube folder and restarts the download again. The code is quite simple:
#!/usr/bin/python
# -*- coding: utf8 -*-
# Modules used:
import sys, os
part_files = os.popen ("ls *.part")
for part_file in part_files:
target_name = part_file.strip()[:-5]
target_name = target_name.split(".")[0]
yt_id = target_name [len(target_name)-11:]
cli = 'youtube-dl -t http://www.youtube.com/watch?v=' + yt_id
print cli
os.popen (cli)
Unfortunately youtube-dl bombs out with the following stack trace and I have got no clue why...
youtube-dl -t http://www.youtube.com/watch?v=XNVaEoQMarg
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/bin/youtube-dl/__main__.py", line 17, in <module>
File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 516, in main
File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 500, in _real_main
File "/usr/bin/youtube-dl/youtube_dl/FileDownloader.py", line 507, in download
File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 92, in extract
File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 87, in initialize
File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 313, in _real_initialize
File "/usr/bin/youtube-dl/youtube_dl/InfoExtractors.py", line 197, in report_lang
File "/usr/bin/youtube-dl/youtube_dl/FileDownloader.py", line 192, in to_screen
IOError: [Errno 32] Broken pipe
Does anybody have an idea why this happens ? Using youtube-dl without the "resume download" script works perfectly though...
TIA,
D$
Last edited by Darksoul71 (2013-06-06 11:00:51)

Answer 1

I think the problem is that you are using os.popen.
The function returns a file object to which the pipe of the new spawned process is connected. If you don't catch that file object, python might garbage collect it. (In addition, your script terminates, so the file object will be garbage collected).
Check out the subprocess module from the standard lib. I think there you can redirect the pipe of the new spawned processs to /dev/null or something.
http://docs.python.org/2/library/subprocess.html