Tabulator: format tables with Python

Project page: http://xyne. I've been using this module If you ever find yourself iterating There's a simple example on Part of a larger table as an Quantity {220} lattice spacing of silicon alpha particle-electron mass ratio alpha particle mass alpha particle mass energy equivalent alpha particle mass energy equivalent in MeV alpha particle mass in u alpha particle molar mass alpha particle-proton mass ratio Angstrom star atomic mass constant atomic mass constant energy equivalent atomic mass constant energy equivalent in MeV atomic mass unit-electron volt relationship atomic mass unit-hartree relationship atomic mass unit-hertz relationship atomic mass unit-inverse meter relationship atomic mass unit-joule relationship atomic mass unit-kelvin relationship atomic mass unit-kilogram relationship archlinux.ca/projects/python3-tabulator/
for ages now but have only now got around to releasing it. It makes it dead-simple to print nicely formatted tables in plaintext and other formats, and I'll add gradually.
lists to find the longest member in order to correctly format the column, then this is for you.
the project page to you get started. Just ask if anything is unclear.
example:
Value Uncertainty Unit
1.920155714e-10 3.2e-18 m
7.2942995361e+3 2.9e-6
6.64465675e-27 2.9e-34 kg
5.97191967e-10 2.6e-17 J
3.727379240e+3 8.2e-5 MeV
4.001506179125e+0 6.2e-11 u
4.001506179125e-3 6.2e-14 kg mol^-1
3.97259968933e+0 3.6e-10
1.00001495e-10 9.0e-17 m
1.660538921e-27 7.3e-35 kg
1.492417954e-10 6.6e-18 J
9.31494061e+2 2.1e-5 MeV
9.31494061e+8 2.1e+1 eV
3.4231776845e+7 2.4e-2 E_h
2.2523427168e+23 1.6e+14 Hz
7.5130066042e+14 5.3e+5 m^-1
1.492417954e-10 6.6e-18 J
1.08095408e+13 9.8e+6 K
1.660538921e-27 7.3e-35 kg

Hi,
There are two workarounds that you could try. 1. if your system could print to pdf, you could print each response as pdf instead of export as pdf.
2. you could duplicate the form with responses, open the dulplicate form, remove some old responses (the one that you already exported) so you don't see the 13000 visible cells dialog, then you could export each response to pdf.
Hope these workarounds could help you while we are getting the fix out.
Regards,
Perry

Similar Messages

How can I format-table with two different variables

I have a script who's purpose is to go through all the mailboxes in exchange and look for any with SG_ or Group listed in the Full Access Permissions. The script works though I'm having trouble formatting the data in a way that I can use it in Excel.
Ultimately I'd like it to report the Name of the mailbox and the name of the entry listed in the Full Access Permissions that has SG_ or Group in the name. Currently the output is the name of the mailbox and then on a separate line the full name of the security
group.
I'm trying to have it so that each part is in it's own column. That will make it easy to review in excel.
$Grabmailbox = Get-Mailbox -Resultsize Unlimited
foreach ($mailbox in $Grabmailbox){
$Getpermission = Get-MailboxPermission $mailbox | where { ($_.User -like "corp\SG_*") -or ($_.User -like "corp\*_group") }
if ($Getpermission -ne $null) {
echo "$mailbox + "$Getpermission | FT User |Out-File c:\Script\VerifiedSG.txt -Append
else{
echo "$mailbox + No SG detected" | Out-File c:\Script\VerifiedSG.txt -Append

Hi Yan,
Thanks for the response but what I am looking for is even simpler than that.
The script goes through every mailbox we have.
It checks the Full Access Permissions for anything with SG_ or group listed in the Full access permissions.
It then should write out to a file the name of the mailbox and the entry in the Full access permissions that contains SG_ or _Group
so something like this:
Mailbox Name <---New ColumN--->Entry
Mailbox1<---New ColumN--->SG_Mailbox1
Ignore the <---New ColumN---> as I simply put that in to emphasize space between the two columns.

Bug in formatting Tables with cel styles + toolbox

We continue to have major problems with the cel styles. Lines disappear due to adjacent cells.
Let me ellaborate:
I have a cel style that has no background and only a .5 line at the bottom.
Underneith I have a cel style that has NO lines. What happens is that when I link that style, the line from the cel above disappears. Then I reselect the top cel and link the correct style again, the line appears.
This is obviously a totally unworkable situation.
What helps 80% of the time, is to go in the options menu for the style. Then give all lines 1pt, black and a style. And then, in exactly this order, delete the linethickness, set the color to ignore and then set the style to ignore (ignore is a translation from Dutch, I do not know the exact term used in the US version). After this I save the style and usually it works.
This bug has been a part of InDesign since the introduction of celstyles.
And while I am ranting, another thing that has been irritating us since the early days. Quite often multiple styles in the panels seem selected. This happens in all panes (paragraph, type, cell and table styles)
It seems to us that Adobe is only paying attention to adding new features, instead of fixing well known bugs.
Does anyone have a solution to my issues?
Any help is well appreciated.

My original post wasn't too clear (sorry). The bug is with TR.
The following code should write the numbers 1 to 10 with four spaces separating each value. But no spaces are written.
program tst2
integer :: i
do i=1,10
write(10,'(I4)',advance='no')i
write(10,'(TR4)',advance='no')
end do
close(10)
end program tst2
In fact the file fort.10 which is produced contains
12345678910
Whereas, with gfortran, open64, ifort, nag Fortran compilers I get (there are four spaces, this forum just doesn't show them all):
1 2 3 4 5 6 7 8 9 10
Edited by: davidb on 16-Oct-2011 15:07

Hello Anybody, I have a question. Can any of you please suggest me how to make an xml file from the database table with all the rows? Note:- I am having the XSD Schema file and the resulted XML file should be in that XSD format only.

Hello Anybody, I have a question. Can any of you please suggest me how to make an xml file from the database table with all the records?
Note:- I am having the XSD Schema file and the resulted XML file should be in that XSD format only.

The Oracle documentation has a good overview of the options available
Generating XML Data from the Database
Without knowing your version, I just picked 11.2, so you made need to look for that chapter in the documentation for your version to find applicable information.
You can also find some information in XML DB FAQ

SSIS 2012 is intermittently failing with below "Invalid date format" while importing data from a source table into a Destination table with same exact schema.

We migrated Packages from SSIS 2008 to 2012. The Package is working fine in all the environments except in one of our environment.
SSIS 2012 is intermittently failing with below error while importing data from a source table into a Destination table with same exact schema.
Error: 2014-01-28 15:52:05.19
   Code: 0x80004005
   Source: xxxxxxxx SSIS.Pipeline
   Description: Unspecified error
End Error
Error: 2014-01-28 15:52:05.19
   Code: 0xC0202009
   Source: Process xxxxxx Load TableName [48]
   Description: SSIS Error Code DTS_E_OLEDBERROR. An OLE DB error has occurred. Error code: 0x80004005.
An OLE DB record is available. Source: "Microsoft SQL Server Native Client 11.0" Hresult: 0x80004005 Description: "Invalid date format".
End Error
Error: 2014-01-28 15:52:05.19
   Code: 0xC020901C
   Source: Process xxxxxxxx Load TableName [48]
   Description: There was an error with Load TableName.Inputs[OLE DB Destination Input].Columns[Updated] on Load TableName.Inputs[OLE DB Destination Input]. The column status returned was: "Conversion failed because the data value overflowed
the specified type.".
End Error
But when we reorder the column in "Updated" in Destination table, the package is importing data successfully.
This looks like bug to me, Any suggestion?

Hi Mohideen,
Based on my research, the issue might be related to one of the following factors:
Memory pressure. Check there is a memory challenge when the issue occurs. In addition, if the package runs in 32-bit runtime on the specific server, use the 64-bit runtime instead.
A known issue with SQL Native Client. As a workaround, use .NET data provider instead of SNAC.
Hope this helps.
Regards,
Mike Yin
If you have any feedback on our support, please click
here
Mike Yin
TechNet Community Support

How to correctly and automatically convert a pdf having tables with embedded figures to MS Word format?

I'm a language translator. My source files are very often pdf's containing huge tables with hundreds of embedded figures as well as text. So far, I've been unable to find a way to automatically convert such a structure to MS Word format. When selecting the table, the figures do not copy. Figures can be selected and copied only one at a time. Tried Reader, tried Acrobat, tried third party software like ABBYY PDF Transformer+ to no avail. Is there a way to do it?

Hi pielassss,
Could you share a sample file with me at [email protected] so that I can check at my end?
What version of Acrobat are you uisng?
Regards,
Rave

LabVIEW binary format that can be opened with Python

I need to save data in a binary format which can be opened with Python. I have tried the "Write Waveforms to File (1D).vi" and can open them back up in LabView, but the binary format is not published anywhere, so I cannot open them in Python. They appear to be saving as DataLog type binary files.
Is there another binary format which I can use that I can open in Python?
The reason I need this is that with 16-channel DAQ at 20kHz I can only save about 30 seconds of data. For longer periods, the task of writing to a text file consumes all available memory and cpu.

I found a solution. I found an open-source project called pyTDMS (google for it), and it can read tdms type files sometimes. It was some trial and error to get my data saved in a way that pyTDMS could open the file, but in the end it works great.
I have two digital output channels as well as 16 analog input channels, so here is how I had to save the data within LabView:
The 16 analog input channels were saved as a 1D array of waveforms. Then the digital output channels were made into another 1D array of waveforms. The trick was to use the write tdms subvi twice. Once for the analog input channels and again, on the same file, for the digital output channels. pyTDMS opens these files just fine, and I can use the metadata to sort out what channels the data goes with in Python.

How to disply internal table with grid format .

HI ,
how to disply internal table with grid format .
Regards
venkat

Grid format can be disaplyed using two ways,
1. Using reuse_alv_grid_display
2. using object oriented ABAP with the methos set_table_for_first_display.
For example program search in where used list for standard SAP programs.
If this is not the answer then please explain your issue in detail
Thanks,
Rama Krishna

Help with Importing Excel Data into Formatted Tables

This is my first post, here, so please be gentle!
I am a relatively new user of InDesign CS4, and I am creating a 70-pg manufacturer's price book. A very large portion of each page is going to be size and price information imported from a large Excel spreadsheet.
I have created the table format that I'd like to use for each page, but the trouble comes when I import the Excel data into that table. For some reason, when I import, it all dumps into one cell. Would it be best to import as an unformatted table, and then format the table each time, or is there a way to simply import the data into my pre-formatted table? I've seen how the former is done, but the latter seems much easier (...although that could be my inexperience talking).
Any advice would be greatly appreciated!
Thanks so much,
Laura (V1500)

Thank you both so much for your time! This is exactly what I needed.
Cheers
Laura

[SOLVED] tv_grab_nl_py works with python 2.6.5, fails on 3.1.2

Hi All,
I have updated my mediacenter. Now tv_grab_nl_py does not work anymore:
[cedric@tv ~]$ tv_grab_nl_py --output ~/listings.xml --fast
File "/usr/bin/tv_grab_nl_py", line 341
print 'tv_grab_nl_py: A grabber that grabs tvguide data from tvgids.nl\n'
^
SyntaxError: invalid syntax
[cedric@tv ~]$
the version of python on the mediacenter (running arch linux):
[cedric@tv ~]$ python
Python 3.1.2 (r312:79147, Oct 4 2010, 12:35:40)
[GCC 4.5.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
I have copied the file to my laptop, there it looks like it's working:
./tv_grab_nl_py --output ~/listings.xml --fast
Config file /home/cedric/.xmltv/tv_grab_nl_py.conf not found.
Re-run me with the --configure flag.
cedric@laptop:~$
the version of python on my laptop (running arch linux):
cedric@laptop:~$ python
Python 2.6.5 (r265:79063, Apr 1 2010, 05:22:20)
[GCC 4.4.3 20100316 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
the script I'm trying to run:
[cedric@tv ~]$ cat tv_grab_nl_py
#!/usr/bin/env python
# $LastChangedDate: 2009-11-14 10:06:41 +0100 (Sat, 14 Nov 2009) $
# $Rev: 104 $
# $Author: pauldebruin $
SYNOPSIS
tv_grab_nl_py is a python script that trawls tvgids.nl for TV
programming information and outputs it in XMLTV-formatted output (see
http://membled.com/work/apps/xmltv). Users of MythTV
(http://www.mythtv.org) will appreciate the output generated by this
grabber, because it fills the category fields, i.e. colors in the EPG,
and has logos for most channels automagically available. Check the
website below for screenshots. The newest version of this script can be
found here:
http://code.google.com/p/tvgrabnlpy/
USAGE
Check the web site above and/or run script with --help and start from there
HISTORY
tv_grab_nl_py used to be called tv_grab_nl_pdb, first released on
2003/07/09. The name change was necessary because more and more people
are actively contributing to this script and I always disliked using my
initials (I was just too lazy to change it). At the same time I switched
from using CVS to SVN and as a result the version numbering scheme has
changed. The lastest official release of tv_grab_nl_pdb is 0.48. The
first official release of tv_grab_nl_py is 6.
QUESTIONS
Questions (and patches) are welcome at: paul at pwdebruin dot net.
IMPORTANT NOTES
If you were using tv_grab_nl from the XMLTV bundle then enable the
compat flag or use the --compat command-line option. Otherwise, the
xmltvid's are wrong and you will not see any new data in MythTV.
CONTRIBUTORS
Main author: Paul de Bruin (paul at pwdebruin dot net)
Michel van der Laan made available his extensive collection of
high-quality logos that is used by this script.
Michael Heus has taken the effort to further enhance this script so that
it now also includes:
- Credit info: directors, actors, presenters and writers
- removal of programs that are actually just groupings/broadcasters
(e.g. "KETNET", "Wild Friday", "Z@pp")
- Star-rating for programs tipped by tvgids.nl
- Black&White, Stereo and URL info
- Better detection of Movies
- and much, much more...
Several other people have provided feedback and patches (these are the
people I could find in my email archive, if you are missing from this
list let me know):
Huub Bouma, Roy van der Kuil, Remco Rotteveel, Mark Wormgoor, Dennis van
Onselen, Hugo van der Kooij, Han Holl, Ian Mcdonald, Udo van den Heuvel.
# Modules we need
import re, urllib2, getopt, sys
import time, random
import htmlentitydefs, os, os.path, pickle
from string import replace, split, strip
from threading import Thread
from xml.sax import saxutils
# Extra check for the datetime module
try:
import datetime
except:
sys.stderr.write('This script needs the datetime module that was introduced in Python version 2.3.\n')
sys.stderr.write('You are running:\n')
sys.stderr.write('%s\n' % sys.version)
sys.exit(1)
# XXX: fix to prevent crashes in Snow Leopard [Robert Klep]
if sys.platform == 'darwin' and sys.version_info[:3] == (2, 6, 1):
try:
urllib2.urlopen('http://localhost.localdomain')
except:
pass
# do extra debug stuff
debug = 1
try:
import redirect
except:
debug = 0
pass
# globals
# compile only one time
r_entity = re.compile(r'&(#x[0-9A-Fa-f]+|#[0-9]+|[A-Za-z]+);')
tvgids = 'http://www.tvgids.nl/'
uitgebreid_zoeken = tvgids + 'zoeken/'
# how many seconds to wait before we timeout on a
# url fetch, 10 seconds seems reasonable
global_timeout = 10
# Wait a random number of seconds between each page fetch.
# We want to be nice and not hammer tvgids.nl (these are the
# friendly people that provide our data...).
# Also, it appears tvgids.nl throttles its output.
# So there, there is not point in lowering these numbers, if you
# are in a hurry, use the (default) fast mode.
nice_time = [1, 2]
# Maximum length in minutes of gaps/overlaps between programs to correct
max_overlap = 10
# Strategy to use for correcting overlapping prgramming:
# 'average' = use average of stop and start of next program
# 'stop' = keep stop time of current program and adjust start time of next program accordingly
# 'start' = keep start time of next program and adjust stop of current program accordingly
# 'none' = do not use any strategy and see what happens
overlap_strategy = 'average'
# Experimental strategy for clumping overlapping programming, all programs that overlap more
# than max_overlap minutes, but less than the length of the shortest program are clumped
# together. Highly experimental and disabled for now.
do_clump = False
# Create a category translation dictionary
# Look in mythtv/themes/blue/ui.xml for all category names
# The keys are the categories used by tvgids.nl (lowercase please)
cattrans = { 'amusement' : 'Talk',
'animatie' : 'Animated',
'comedy' : 'Comedy',
'documentaire' : 'Documentary',
'educatief' : 'Educational',
'erotiek' : 'Adult',
'film' : 'Film',
'muziek' : 'Art/Music',
'informatief' : 'Educational',
'jeugd' : 'Children',
'kunst/cultuur' : 'Arts/Culture',
'misdaad' : 'Crime/Mystery',
'muziek' : 'Music',
'natuur' : 'Science/Nature',
'nieuws/actualiteiten' : 'News',
'overige' : 'Unknown',
'religieus' : 'Religion',
'serie/soap' : 'Drama',
'sport' : 'Sports',
'theater' : 'Arts/Culture',
'wetenschap' : 'Science/Nature'}
# Create a role translation dictionary for the xmltv credits part
# The keys are the roles used by tvgids.nl (lowercase please)
roletrans = {'regie' : 'director',
'acteurs' : 'actor',
'presentatie' : 'presenter',
'scenario' : 'writer'}
# We have two sources of logos, the first provides the nice ones, but is not
# complete. We use the tvgids logos to fill the missing bits.
logo_provider = [ 'http://visualisation.tudelft.nl/~paul/logos/gif/64x64/',
'http://static.tvgids.nl/gfx/zenders/' ]
logo_names = {
1 : [0, 'ned1'],
2 : [0, 'ned2'],
3 : [0, 'ned3'],
4 : [0, 'rtl4'],
5 : [0, 'een'],
6 : [0, 'canvas_color'],
7 : [0, 'bbc1'],
8 : [0, 'bbc2'],
9 : [0,'ard'],
10 : [0,'zdf'],
11 : [1, 'rtl'],
12 : [0, 'wdr'],
13 : [1, 'ndr'],
14 : [1, 'srsudwest'],
15 : [1, 'rtbf1'],
16 : [1, 'rtbf2'],
17 : [0, 'tv5'],
18 : [0, 'ngc'],
19 : [1, 'eurosport'],
20 : [1, 'tcm'],
21 : [1, 'cartoonnetwork'],
24 : [0, 'canal+red'],
25 : [0, 'mtv-color'],
26 : [0, 'cnn'],
27 : [0, 'rai'],
28 : [1, 'sat1'],
29 : [0, 'discover-spacey'],
31 : [0, 'rtl5'],
32 : [1, 'trt'],
34 : [0, 'veronica'],
35 : [0, 'tmf'],
36 : [0, 'sbs6'],
37 : [0, 'net5'],
38 : [1, 'arte'],
39 : [0, 'canal+blue'],
40 : [0, 'at5'],
46 : [0, 'rtl7'],
49 : [1, 'vtm'],
50 : [1, '3sat'],
58 : [1, 'pro7'],
59 : [1, 'kanaal2'],
60 : [1, 'vt4'],
65 : [0, 'animal-planet'],
73 : [1, 'mezzo'],
86 : [0, 'bbc-world'],
87 : [1, 'tve'],
89 : [1, 'nick'],
90 : [1, 'bvn'],
91 : [0, 'comedy_central'],
92 : [0, 'rtl8'],
99 : [1, 'sport1_1'],
100 : [0, 'rtvu'],
101 : [0, 'tvwest'],
102 : [0, 'tvrijnmond'],
103 : [1, 'tvnoordholland'],
104 : [1, 'bbcprime'],
105 : [1, 'spiceplatinum'],
107 : [0, 'canal+yellow'],
108 : [0, 'tvnoord'],
109 : [0, 'omropfryslan'],
114 : [0, 'omroepbrabant']}
# A selection of user agents we will impersonate, in an attempt to be less
# conspicuous to the tvgids.nl police.
user_agents = [ 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)',
'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)',
'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)',
'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.9) Gecko/20071105 Firefox/2.0.0.9',
'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9',
'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.8) Gecko/20071022 Ubuntu/7.10 (gutsy) Firefox/2.0.0.8'
# Work in progress, the idea is to cache program categories and
# descriptions to eliminate a lot of page fetches from tvgids.nl
# for programs that do not have interesting/changing descriptions
class ProgramCache:
A cache to hold program name and category info.
TVgids stores the detail for each program on a separate URL with an
(apparently unique) ID. This cache stores the fetched info with the ID.
New fetches will use the cached info instead of doing an (expensive)
page fetch.
def __init__(self, filename=None):
Create a new ProgramCache object, optionally from file
# where we store our info
self.filename = filename
if filename == None:
self.pdict = {}
else:
if os.path.isfile(filename):
self.load(filename)
else:
self.pdict = {}
def load(self, filename):
Loads a pickled cache dict from file
try:
self.pdict = pickle.load(open(filename,'r'))
except:
sys.stderr.write('Error loading cache file: %s (possibly corrupt)' % filename)
sys.exit(2)
def dump(self, filename):
Dumps a pickled cache, and makes sure it is valid
if os.access(filename, os.F_OK):
try:
os.remove(filename)
except:
sys.stderr.write('Cannot remove %s, check permissions' % filename)
pickle.dump(self.pdict, open(filename+'.tmp', 'w'))
os.rename(filename+'.tmp', filename)
def query(self, program_id):
Updates/gets/whatever.
try:
return self.pdict[program_id]
except:
return None
def add(self, program):
Adds a program
self.pdict[program['ID']] = program
def clear(self):
Clears the cache (i.e. empties it)
self.pdict = {}
def clean(self):
Removes all cached programming before today.
Also removes erroneously cached programming.
now = time.localtime()
dnow = datetime.datetime(now[0],now[1],now[2])
for key in self.pdict.keys():
try:
if self.pdict[key]['stop-time'] < dnow or self.pdict[key]['name'].lower() == 'onbekend':
del self.pdict[key]
except:
pass
def usage():
print 'tv_grab_nl_py: A grabber that grabs tvguide data from tvgids.nl\n'
print 'and stores it in XMLTV-combatible format.\n'
print 'Usage:'
print '--help, -h = print this info'
print '--configure = create configfile (overwrites existing file)'
print '--config-file = name of the configuration file (default = ~/.xmltv/tv_grab_py.conf'
print '--capabilities = xmltv required option'
print '--desc-length = maximum allowed length of programme descriptions in bytes.'
print '--description = prints a short description of the grabber'
print '--output = file where to put the output'
print '--days = # number of days to grab'
print '--preferredmethod = returns the preferred method to be called'
print '--fast = do not grab descriptions of programming'
print '--slow = grab descriptions of programming'
print '--quiet = suppress all output'
print '--compat = append tvgids.nl to the xmltv id (use this if you were using tv_grab_nl)'
print '--logos 0/1 = insert urls to channel icons (mythfilldatabase will then use these)'
print '--nocattrans = do not translate the grabbed genres into MythTV-genres'
print '--cache = cache descriptions and use the file to store'
print '--clean_cache = clean the cache file before fetching'
print '--clear_cache = empties the cache file before fetching data'
print '--slowdays = grab slowdays initial days and the rest in fast mode'
print '--max_overlap = maximum length of overlap between programming to correct [minutes]'
print '--overlap_strategy = what strategy to use to correct overlaps (check top of source code)'
def filter_line_identity(m, defs=htmlentitydefs.entitydefs):
# callback: translate one entity to its ISO Latin value
k = m.group(1)
if k.startswith("#") and k[1:] in xrange(256):
return chr(int(k[1:]))
try:
return defs[k]
except KeyError:
return m.group(0) # use as is
def filter_line(s):
Removes unwanted stuff in strings (adapted from tv_grab_be)
# do the latin1 stuff
s = r_entity.sub(filter_line_identity, s)
s = replace(s,' ',' ')
# Ik vermoed dat de volgende drie regels overbodig zijn, maar ze doen
# niet veel kwaad -- Han Holl
s = replace(s,'\r',' ')
x = re.compile('(<.*?>)') # Udo
s = x.sub('', s) #Udo
s = replace(s, '~Q', "'")
s = replace(s, '~R', "'")
# Hmm, not sure if I understand this. Without it, mythfilldatabase barfs
# on program names like "Steinbrecher &..."
# We most create valid XML -- Han Holl
s = saxutils.escape(s)
return s
def calc_timezone(t):
Takes a time from tvgids.nl and formats it with all the required
timezone conversions.
in: '20050429075000'
out:'20050429075000 (CET|CEST)'
Until I have figured out how to correctly do timezoning in python this method
will bork if you are not in a zone that has the same DST rules as 'Europe/Amsterdam'.
year = int(t[0:4])
month = int(t[4:6])
day = int(t[6:8])
hour = int(t[8:10])
minute = int(t[10:12])
#td = {'CET': '+0100', 'CEST': '+0200'}
#td = {'CET': '+0100', 'CEST': '+0200', 'W. Europe Standard Time' : '+0100', 'West-Europa (standaardtijd)' : '+0100'}
td = {0 : '+0100', 1 : '+0200'}
pt = time.mktime((year,month,day,hour,minute,0,0,0,-1))
timezone=''
try:
#timezone = time.tzname[(time.localtime(pt))[-1]]
timezone = (time.localtime(pt))[-1]
except:
sys.stderr.write('Cannot convert time to timezone')
return t+' %s' % td[timezone]
def format_timezone(td):
Given a datetime object, returns a string in XMLTV format
tstr = td.strftime('%Y%m%d%H%M00')
return calc_timezone(tstr)
def get_page_internal(url, quiet=0):
Retrieves the url and returns a string with the contents.
Optionally, returns None if processing takes longer than
the specified number of timeout seconds.
txtdata = None
txtheaders = {'Keep-Alive' : '300',
'User-Agent' : user_agents[random.randint(0, len(user_agents)-1)] }
try:
#fp = urllib2.urlopen(url)
rurl = urllib2.Request(url, txtdata, txtheaders)
fp = urllib2.urlopen(rurl)
lines = fp.readlines()
page = "".join(lines)
return page
except:
if not quiet:
sys.stderr.write('Cannot open url: %s\n' % url)
return None
class FetchURL(Thread):
A simple thread to fetch a url with a timeout
def __init__ (self, url, quiet=0):
Thread.__init__(self)
self.quiet = quiet
self.url = url
self.result = None
def run(self):
self.result = get_page_internal(self.url, self.quiet)
def get_page(url, quiet=0):
Wrapper around get_page_internal to catch the
timeout exception
try:
fu = FetchURL(url, quiet)
fu.start()
fu.join(global_timeout)
return fu.result
except:
if not quiet:
sys.stderr.write('get_page timed out on (>%s s): %s\n' % (global_timeout, url))
return None
def get_channels(file, quiet=0):
Get a list of all available channels and store these
in a file.
# store channels in a dict
channels = {}
# tvgids stores several instances of channels, we want to
# find all the possibile channels
channel_get = re.compile('<optgroup label=.*?>(.*?)</optgroup>', re.DOTALL)
# this is how we will find a (number, channel) instance
channel_re = re.compile('<option value="([0-9]+)" >(.*?)</option>', re.DOTALL)
# this is where we will try to find our channel list
total = get_page(uitgebreid_zoeken, quiet)
if total == None:
return
# get a list of match objects of all the <select blah station>
stations = channel_get.finditer(total)
# and create a dict of number, channel_name pairs
# we do this this way because several instances of the
# channel list are stored in the url and not all of the
# instances have all the channels, this way we get them all.
for station in stations:
m = channel_re.finditer(station.group(0))
for p in m:
try:
a = int(p.group(1))
b = filter_line(p.group(2))
channels[a] = b
except:
sys.stderr.write('Oops, [%s,%s] does not look like a valid channel, skipping it...\n' % (p.group(1),p.group(2)))
# sort on channel number (arbitrary but who cares)
keys = channels.keys()
keys.sort()
# and create a file with the channels
f = open(file,'w')
for k in keys:
f.write("%s %s\n" % (k, channels[k]))
f.close()
def get_channel_all_days(channel, days, quiet=0):
Get all available days of programming for channel number
The output is a list of programming in order where each row
contains a dictionary with program information.
now = datetime.datetime.now()
programs = []
# Tvgids shows programs per channel per day, so we loop over the number of days
# we are required to grab
for offset in range(0, days):
channel_url = 'http://www.tvgids.nl/zoeken/?d=%i&z=%s' % (offset, channel)
# For historic purposes, the old style url that gave us a full week in advance:
# channel_url = 'http://www.tvgids.nl/zoeken/?trefwoord=Titel+of+trefwoord&interval=0&timeslot='+\
# '&station=%s&periode=%i&genre=&order=0' % (channel,days-1)
# Sniff, we miss you...
if offset > 0:
time.sleep(random.randint(nice_time[0], nice_time[1]))
# get the raw programming for the day
total = get_page(channel_url, quiet)
if total == None:
return programs
# Setup a number of regexps
# checktitle will match the title row in H2 tags of the daily overview page, e.g.
# <h2>zondag 19 oktober 2008</h2>
checktitle = re.compile('<h2>(.*?)</h2>',re.DOTALL)
# getrow will locate each row with program details
getrow = re.compile('<a href="/programma/(.*?)</a>',re.DOTALL)
# parserow matches the required program info, with groups:
# 1 = program ID
# 2 = broadcast times
# 3 = program name
parserow = re.compile('(.*?)/.*<span class="time">(.*?)</span>.*<span class="title">(.*?)</span>', re.DOTALL)
# normal begin and end times
times = re.compile('([0-9]+:[0-9]+) - ([0-9]+:[0-9]+)?')
# Get the day of month listed on the page as well as the expected date we are grabbing and compare these.
# If these do not match, we skip parsing the programs on the page and issue a warning.
#dayno = int(checkday.search(total).group(1))
title = checktitle.search(total)
if title:
title = title.group(1)
dayno = title.split()[1]
else:
sys.stderr.write('\nOops, there was a problem with page %s. Skipping it...\n' % (channel_url))
continue
expected = now + datetime.timedelta(days=offset)
if (not dayno.isdigit() or int(dayno) != expected.day):
sys.stderr.write('\nOops, did not expect page %s to list programs for "%s", skipping it...\n' % (channel_url,title))
continue
# and find relevant programming info
allrows = getrow.finditer(total)
for r in allrows:
detail = parserow.search(r.group(1))
if detail != None:
# default times
start_time = None
stop_time = None
# parse for begin and end times
t = times.search(detail.group(2))
if t != None:
start_time = t.group(1)
stop_time = t.group(2)
program_url = 'http://www.tvgids.nl/programma/' + detail.group(1) + '/'
program_name = detail.group(3)
# store time, name and detail url in a dictionary
tdict = {}
tdict['start'] = start_time
tdict['stop'] = stop_time
tdict['name'] = program_name
if tdict['name'] == '':
tdict['name'] = 'onbekend'
tdict['url'] = program_url
tdict['ID'] = detail.group(1)
tdict['offset'] = offset
#Add star rating if tipped by tvgids.nl
tdict['star-rating'] = '';
if r.group(1).find('Tip') != -1:
tdict['star-rating'] = '4/5'
# and append the program to the list of programs
programs.append(tdict)
# done
return programs
def make_daytime(time_string, offset=0, cutoff='00:00', stoptime=False):
Given a string '11:35' and an offset from today,
return a datetime object. The cuttoff specifies the point where the
new day starts.
Examples:
In [2]:make_daytime('11:34',0)
Out[2]:datetime.datetime(2006, 8, 3, 11, 34)
In [3]:make_daytime('11:34',1)
Out[3]:datetime.datetime(2006, 8, 4, 11, 34)
In [7]:make_daytime('11:34',0,'12:00')
Out[7]:datetime.datetime(2006, 8, 4, 11, 34)
In [4]:make_daytime('11:34',0,'11:34',False)
Out[4]:datetime.datetime(2006, 8, 3, 11, 34)
In [5]:make_daytime('11:34',0,'11:34',True)
Out[5]:datetime.datetime(2006, 8, 4, 11, 34)
h,m = [int(x) for x in time_string.split(':')];
hm = int(time_string.replace(':',''))
chm = int(cutoff.replace(':',''))
# check for the cutoff, if the time is before the cutoff then
# add a day
extra_day = 0
if (hm < chm) or (stoptime==True and hm == chm):
extra_day = 1
# and create a datetime object, DST is handled at a later point
pt = time.localtime()
dt = datetime.datetime(pt[0],pt[1],pt[2],h,m)
dt = dt + datetime.timedelta(offset+extra_day)
return dt
def correct_times(programs, quiet=0):
Parse a list of programs as generated by get_channel_all_days() and
convert begin and end times to xmltv compatible times in datetime objects.
if programs == []:
return programs
# the start time of programming for this day, times *before* this time are
# assumed to be on the next day
day_start_time = '06:00'
# initialise using the start time of the first program on this day
if programs[0]['start'] != None:
day_start_time = programs[0]['start']
for program in programs:
if program['start'] == program['stop']:
program['stop'] = None
# convert the times
if program['start'] != None:
program['start-time'] = make_daytime(program['start'], program['offset'], day_start_time)
else:
program['start-time'] = None
if program['stop'] != None:
program['stop-time'] = make_daytime(program['stop'], program['offset'], day_start_time, stoptime=True)
# extra correction, needed because the stop time of a program may be on the next day, after the
# day cutoff. For example:
# 06:00 - 23:40 Long Program
# 23:40 - 00:10 Lala
# 00:10 - 08:00 Wawa
# This puts the end date of Wawa on the current, instead of the next day. There is no way to detect
# this with a single cutoff in make_daytime. Therefore, check if there is a day difference between
# start and stop dates and correct if necessary.
if program['start-time'] != None:
# make two dates
start = program['start-time']
stop = program['stop-time']
single_day = datetime.timedelta(1)
startdate = datetime.datetime(start.year,start.month,start.day)
stopdate = datetime.datetime(stop.year,stop.month,stop.day)
if startdate - stopdate == single_day:
program['stop-time'] = program['stop-time'] + single_day
else:
program['stop-time'] = None
def parse_programs(programs, offset=0, quiet=0):
Parse a list of programs as generated by get_channel_all_days() and
convert begin and end times to xmltv compatible times.
# good programs
good_programs = []
# calculate absolute start and stop times
correct_times(programs, quiet)
# next, correct for missing end time and copy over all good programming to the
# good_programs list
for i in range(len(programs)):
# Try to correct missing end time by taking start time from next program on schedule
if (programs[i]['stop-time'] == None and i < len(programs)-1):
if not quiet:
sys.stderr.write('Oops, "%s" has no end time. Trying to fix...\n' % programs[i]['name'])
programs[i]['stop-time'] = programs[i+1]['start-time']
# The common case: start and end times are present and are not
# equal to each other (yes, this can happen)
if programs[i]['start-time'] != None and \
programs[i]['stop-time'] != None and \
programs[i]['start-time'] != programs[i]['stop-time']:
good_programs.append(programs[i])
# Han Holl: try to exclude programs that stop before they begin
for i in range(len(good_programs)-1,-1,-1):
if good_programs[i]['stop-time'] <= good_programs[i]['start-time']:
if not quiet:
sys.stderr.write('Deleting invalid stop/start time: %s\n' % good_programs[i]['name'])
del good_programs[i]
# Try to exclude programs that only identify a group or broadcaster and have overlapping start/end times with
# the actual programs
for i in range(len(good_programs)-2,-1,-1):
if good_programs[i]['start-time'] <= good_programs[i+1]['start-time'] and \
good_programs[i]['stop-time'] >= good_programs[i+1]['stop-time']:
if not quiet:
sys.stderr.write('Deleting grouping/broadcaster: %s\n' % good_programs[i]['name'])
del good_programs[i]
for i in range(len(good_programs)-1):
# PdB: Fix tvgids start-before-end x minute interval overlap. An overlap (positive or
# negative) is halved and each half is assigned to the adjacent programmes. The maximum
# overlap length between programming is set by the global variable 'max_overlap' and is
# default 10 minutes. Examples:
# Positive overlap (= overlap in programming):
# 10:55 - 12:00 Lala
# 11:55 - 12:20 Wawa
# is transformed in:
# 10:55 - 11.57 Lala
# 11:57 - 12:20 Wawa
# Negative overlap (= gap in programming):
# 10:55 - 11:50 Lala
# 12:00 - 12:20 Wawa
# is transformed in:
# 10:55 - 11.55 Lala
# 11:55 - 12:20 Wawa
stop = good_programs[i]['stop-time']
start = good_programs[i+1]['start-time']
dt = stop-start
avg = start + dt / 2
overlap = 24*60*60*dt.days + dt.seconds
# check for the size of the overlap
if 0 < abs(overlap) <= max_overlap*60:
if not quiet:
if overlap > 0:
sys.stderr.write('"%s" and "%s" overlap %s minutes. Adjusting times.\n' % \
(good_programs[i]['name'],good_programs[i+1]['name'],overlap / 60))
else:
sys.stderr.write('"%s" and "%s" have gap of %s minutes. Adjusting times.\n' % \
(good_programs[i]['name'],good_programs[i+1]['name'],abs(overlap) / 60))
# stop-time of previous program wins
if overlap_strategy == 'stop':
good_programs[i+1]['start-time'] = good_programs[i]['stop-time']
# start-time of next program wins
elif overlap_strategy == 'start':
good_programs[i]['stop-time'] = good_programs[i+1]['start-time']
# average the difference
elif overlap_strategy == 'average':
good_programs[i]['stop-time'] = avg
good_programs[i+1]['start-time'] = avg
# leave as is
else:
pass
# Experimental strategy to make sure programming does not disappear. All programs that overlap more
# than the maximum overlap length, but less than the shortest length of the two programs are
# clumped.
if do_clump:
for i in range(len(good_programs)-1):
stop = good_programs[i]['stop-time']
start = good_programs[i+1]['start-time']
dt = stop-start
overlap = 24*60*60*dt.days + dt.seconds
length0 = good_programs[i]['stop-time'] - good_programs[i]['start-time']
length1 = good_programs[i+1]['stop-time'] - good_programs[i+1]['start-time']
l0 = length0.days*24*60*60 + length0.seconds
l1 = length1.days*24*60*60 + length0.seconds
if abs(overlap) >= max_overlap*60 <= min(l0,l1)*60 and \
not good_programs[i].has_key('clumpidx') and \
not good_programs[i+1].has_key('clumpidx'):
good_programs[i]['clumpidx'] = '0/2'
good_programs[i+1]['clumpidx'] = '1/2'
good_programs[i]['stop-time'] = good_programs[i+1]['stop-time']
good_programs[i+1]['start-time'] = good_programs[i]['start-time']
# done, nothing to see here, please move on
return good_programs
def get_descriptions(programs, program_cache=None, nocattrans=0, quiet=0, slowdays=0):
Given a list of programs, from get_channel, retrieve program information
# This regexp tries to find details such as Genre, Acteurs, Jaar van Premiere etc.
detail = re.compile('<li>.*?<strong>(.*?):</strong>.*?<br />(.*?)</li>', re.DOTALL)
# These regexps find the description area, the program type and descriptive text
description = re.compile('<div class="description">.*?<div class="text"(.*?)<div class="clearer"></div>',re.DOTALL)
descrtype = re.compile('<div class="type">(.*?)</div>',re.DOTALL)
descrline = re.compile('<p>(.*?)</p>',re.DOTALL)
# randomize detail requests
nprograms = len(programs)
fetch_order = range(0,nprograms)
random.shuffle(fetch_order)
counter = 0
for i in fetch_order:
counter += 1
if programs[i]['offset'] >= slowdays:
continue
if not quiet:
sys.stderr.write('\n(%3.0f%%) %s: %s ' % (100*float(counter)/float(nprograms), i, programs[i]['name']))
# check the cache for this program's ID
cached_program = program_cache.query(programs[i]['ID'])
if (cached_program != None):
if not quiet:
sys.stderr.write(' [cached]')
# copy the cached information, except the start/end times, rating and clumping,
# these may have changed.
tstart = programs[i]['start-time']
tstop = programs[i]['stop-time']
rating = programs[i]['star-rating']
try:
clump = programs[i]['clumpidx']
except:
clump = False
programs[i] = cached_program
programs[i]['start-time'] = tstart
programs[i]['stop-time'] = tstop
programs[i]['star-rating'] = rating
if clump:
programs[i]['clumpidx'] = clump
continue
else:
# be nice to tvgids.nl
time.sleep(random.randint(nice_time[0], nice_time[1]))
# get the details page, and get all the detail nodes
descriptions = ()
details = ()
try:
if not quiet:
sys.stderr.write(' [normal fetch]')
total = get_page(programs[i]['url'])
details = detail.finditer(total)
descrspan = description.search(total);
descriptions = descrline.finditer(descrspan.group(1))
except:
# if we cannot find the description page,
# go to next in the loop
if not quiet:
sys.stderr.write(' [fetch failed or timed out]')
continue
# define containers
programs[i]['credits'] = {}
programs[i]['video'] = {}
# now parse the details
line_nr = 1;
# First, we try to find the program type in the description section.
# Note that this is not the same as the generic genres (these are searched later on), but a more descriptive one like "Culinair programma"
# If present, we store this as first part of the regular description:
programs[i]['detail1'] = descrtype.search(descrspan.group(1)).group(1).capitalize()
if programs[i]['detail1'] != '':
line_nr = line_nr + 1
# Secondly, we add one or more lines of the program description that are present.
for descript in descriptions:
d_str = 'detail' + str(line_nr)
programs[i][d_str] = descript.group(1)
# Remove sponsored link from description if present.
sponsor_pos = programs[i][d_str].rfind('<i>Gesponsorde link:</i>')
if sponsor_pos > 0:
programs[i][d_str] = programs[i][d_str][0:sponsor_pos]
programs[i][d_str] = filter_line(programs[i][d_str]).strip()
line_nr = line_nr + 1
# Finally, we check out all program details. These are generically denoted as:
# <li><strong>(TYPE):</strong><br />(CONTENT)</li>
# Some examples:
# <li><strong>Genre:</strong><br />16 oktober 2008</li>
# <li><strong>Genre:</strong><br />Amusement</li>
for d in details:
type = d.group(1).strip().lower()
content_asis = d.group(2).strip()
content = filter_line(content_asis).strip()
if content == '':
continue
elif type == 'genre':
# Fix detection of movies based on description as tvgids.nl sometimes
# categorises a movie as e.g. "Komedie", "Misdaadkomedie", "Detectivefilm".
genre = content;
if (programs[i]['detail1'].lower().find('film') != -1 \
or programs[i]['detail1'].lower().find('komedie') != -1)\
and programs[i]['detail1'].lower().find('tekenfilm') == -1 \
and programs[i]['detail1'].lower().find('animatiekomedie') == -1 \
and programs[i]['detail1'].lower().find('filmpje') == -1:
genre = 'film'
if nocattrans:
programs[i]['genre'] = genre.title()
else:
try:
programs[i]['genre'] = cattrans[genre.lower()]
except:
programs[i]['genre'] = ''
# Parse persons and their roles for credit info
elif roletrans.has_key(type):
programs[i]['credits'][roletrans[type]] = []
persons = content_asis.split(',');
for name in persons:
if name.find(':') != -1:
name = name.split(':')[1]
if name.find('-') != -1:
name = name.split('-')[0]
if name.find('e.a') != -1:
name = name.split('e.a')[0]
programs[i]['credits'][roletrans[type]].append(filter_line(name.strip()))
elif type == 'bijzonderheden':
if content.find('Breedbeeld') != -1:
programs[i]['video']['breedbeeld'] = 1
if content.find('Zwart') != -1:
programs[i]['video']['blackwhite'] = 1
if content.find('Teletekst') != -1:
programs[i]['teletekst'] = 1
if content.find('Stereo') != -1:
programs[i]['stereo'] = 1
elif type == 'url':
programs[i]['infourl'] = content
else:
# In unmatched cases, we still add the parsed type and content to the program details.
# Some of these will lead to xmltv output during the xmlefy_programs step
programs[i][type] = content
# do not cache programming that is unknown at the time
# of fetching.
if programs[i]['name'].lower() != 'onbekend':
program_cache.add(programs[i])
if not quiet:
sys.stderr.write('\ndone...\n\n')
# done
def title_split(program):
Some channels have the annoying habit of adding the subtitle to the title of a program.
This function attempts to fix this, by splitting the name at a ': '.
if (program.has_key('titel aflevering') and program['titel aflevering'] != '') \
or (program.has_key('genre') and program['genre'].lower() in ['movies','film']):
return
colonpos = program['name'].rfind(': ')
if colonpos > 0:
program['titel aflevering'] = program['name'][colonpos+1:len(program['name'])].strip()
program['name'] = program['name'][0:colonpos].strip()
def xmlefy_programs(programs, channel, desc_len, compat=0, nocattrans=0):
Given a list of programming (from get_channels())
returns a string with the xml equivalent
output = []
for program in programs:
clumpidx = ''
try:
if program.has_key('clumpidx'):
clumpidx = 'clumpidx="'+program['clumpidx']+'"'
except:
print program
output.append(' <programme start="%s" stop="%s" channel="%s%s" %s> \n' % \
(format_timezone(program['start-time']), format_timezone(program['stop-time']),\
channel, compat and '.tvgids.nl' or '', clumpidx))
output.append(' <title lang="nl">%s</title>\n' % filter_line(program['name']))
if program.has_key('titel aflevering') and program['titel aflevering'] != '':
output.append(' <sub-title lang="nl">%s</sub-title>\n' % filter_line(program['titel aflevering']))
desc = []
for detail_row in ['detail1','detail2','detail3']:
if program.has_key(detail_row) and not re.search('[Gg]een detailgegevens be(?:kend|schikbaar)', program[detail_row]):
desc.append('%s ' % program[detail_row])
if desc != []:
# join and remove newlines from descriptions
desc_line = "".join(desc).strip()
desc_line.replace('\n', ' ')
if len(desc_line) > desc_len:
spacepos = desc_line[0:desc_len-3].rfind(' ')
desc_line = desc_line[0:spacepos] + '...'
output.append(' <desc lang="nl">%s</desc>\n' % desc_line)
# Process credits section if present.
# This will generate director/actor/presenter info.
if program.has_key('credits') and program['credits'] != {}:
output.append(' <credits>\n')
for role in program['credits']:
for name in program['credits'][role]:
if name != '':
output.append(' <%s>%s</%s>\n' % (role, name, role))
output.append(' </credits>\n')
if program.has_key('jaar van premiere') and program['jaar van premiere'] != '':
output.append(' <date>%s</date>\n' % program['jaar van premiere'])
if program.has_key('genre') and program['genre'] != '':
output.append(' <category')
if nocattrans:
output.append(' lang="nl"')
output.append ('>%s</category>\n' % program['genre'])
if program.has_key('infourl') and program['infourl'] != '':
output.append(' <url>%s</url>\n' % program['infourl'])
if program.has_key('aflevering') and program['aflevering'] != '':
output.append(' <episode-num system="onscreen">%s</episode-num>\n' % filter_line(program['aflevering']))
# Process video section if present
if program.has_key('video') and program['video'] != {}:
output.append(' <video>\n');
if program['video'].has_key('breedbeeld'):
output.append(' <aspect>16:9</aspect>\n')
if program['video'].has_key('blackwhite'):
output.append(' <colour>no</colour>\n')
output.append(' </video>\n')
if program.has_key('stereo'):
output.append(' <audio><stereo>stereo</stereo></audio>\n')
if program.has_key('teletekst'):
output.append(' <subtitles type="teletext" />\n')
# Set star-rating if applicable
if program['star-rating'] != '':
output.append(' <star-rating><value>%s</value></star-rating>\n' % program['star-rating'])
output.append(' </programme>\n')
return "".join(output)
def main():
# Parse command line options
try:
opts, args = getopt.getopt(sys.argv[1:], "h", ["help", "output=", "capabilities",
"preferredmethod", "days=",
"configure", "fast", "slow",
"cache=", "clean_cache",
"slowdays=","compat",
"desc-length=","description",
"nocattrans","config-file=",
"max_overlap=", "overlap_strategy=",
"clear_cache", "quiet","logos="])
except getopt.GetoptError:
usage()
sys.exit(2)
# DEFAULT OPTIONS - Edit if you know what you are doing
# where the output goes
output = None
output_file = None
# the total number of days to fetch
days = 6
# Fetch data in fast mode, i.e. do NOT grab all the detail information,
# fast means fast, because as it then does not have to fetch a web page for each program
# Default: fast=0
fast = 0
# number of days to fetch in slow mode. For example: --days 5 --slowdays 2, will
# fetch the first two days in slow mode (with all the details) and the remaining three
# days in fast mode.
slowdays = 6
# no output
quiet = 0
# insert url of channel logo into the xml data, this will be picked up by mythfilldatabase
logos = 1
# enable this option if you were using tv_grab_nl, it adjusts the generated
# xmltvid's so that everything works.
compat = 0
# enable this option if you do not want the tvgids categories being translated into
# MythTV-categories (genres)
nocattrans = 0
# Maximum number of characters to use for program description.
# Different values may work better in different versions of MythTV.
desc_len = 475
# default configuration file locations
hpath = ''
if os.environ.has_key('HOME'):
hpath = os.environ['HOME']
# extra test for windows users
elif os.environ.has_key('HOMEPATH'):
hpath = os.environ['HOMEPATH']
# hpath = ''
xmltv_dir = hpath+'/.xmltv'
program_cache_file = xmltv_dir+'/program_cache'
config_file = xmltv_dir+'/tv_grab_nl_py.conf'
# cache the detail information.
program_cache = None
clean_cache = 1
clear_cache = 0
# seed the random generator
random.seed(time.time())
for o, a in opts:
if o in ("-h", "--help"):
usage()
sys.exit(1)
if o == "--quiet":
quiet = 1;
if o == "--description":
print "The Netherlands (tv_grab_nl_py $Rev: 104 $)"
sys.exit(0)
if o == "--capabilities":
print "baseline"
print "cache"
print "manualconfig"
print "preferredmethod"
sys.exit(0)
if o == '--preferredmethod':
print 'allatonce'
sys.exit(0)
if o == '--desc-length':
# Use the requested length for programme descriptions.
desc_len = int(a)
if not quiet:
sys.stderr.write('Using description length: %d\n' % desc_len)
for o, a in opts:
if o == "--config-file":
# use the provided name for configuration
config_file = a
if not quiet:
sys.stderr.write('Using config file: %s\n' % config_file)
for o, a in opts:
if o == "--configure":
# check for the ~.xmltv dir
if not os.path.exists(xmltv_dir):
if not quiet:
sys.stderr.write('You do not have the ~/.xmltv directory,')
sys.stderr.write('I am going to make a shiny new one for you...')
os.mkdir(xmltv_dir)
if not quiet:
sys.stderr.write('Creating config file: %s\n' % config_file)
get_channels(config_file)
sys.exit(0)
if o == "--days":
# limit days to maximum supported by tvgids.nl
days = min(int(a),6)
if o == "--compat":
compat = 1
if o == "--nocattrans":
nocattrans = 1
if o == "--fast":
fast = 1
if o == "--output":
output_file = a
try:
output = open(output_file,'w')
# and redirect output
if debug:
debug_file = open('/tmp/kaas.xml','w')
blah = redirect.Tee(output, debug_file)
sys.stdout = blah
else:
sys.stdout = output
except:
if not quiet:
sys.stderr.write('Cannot write to outputfile: %s\n' % output_file)
sys.exit(2)
if o == "--slowdays":
# limit slowdays to maximum supported by tvgids.nl
slowdays = min(int(a),6)
# slowdays implies fast == 0
fast = 0
if o == "--logos":
logos = int(a)
if o == "--clean_cache":
clean_cache = 1
if o == "--clear_cache":
clear_cache = 1
if o == "--cache":
program_cache_file = a
if o == "--max_overlap":
max_overlap = int(a)
if o == "--overlap_strategy":
overlap_strategy = a
# get configfile if available
try:
f = open(config_file,'r')
except:
sys.stderr.write('Config file %s not found.\n' % config_file)
sys.stderr.write('Re-run me with the --configure flag.\n')
sys.exit(1)
#check for cache
program_cache = ProgramCache(program_cache_file)
if clean_cache != 0:
program_cache.clean()
if clear_cache != 0:
program_cache.clear()
# Go!
channels = {}
# Read the channel stuff
for blah in f.readlines():
blah = blah.lstrip()
blah = blah.replace('\n','')
if blah:
if blah[0] != '#':
channel = blah.split()
channels[channel[0]] = " ".join(channel[1:])
# channels are now in channels dict keyed on channel id
# print header stuff
print '<?xml version="1.0" encoding="ISO-8859-1"?>'
print '<!DOCTYPE tv SYSTEM "xmltv.dtd">'
print '<tv generator-info-name="tv_grab_nl_py $Rev: 104 $">'
# first do the channel info
for key in channels.keys():
print ' <channel id="%s%s">' % (key, compat and '.tvgids.nl' or '')
print ' <display-name lang="nl">%s</display-name>' % channels[key]
if (logos):
ikey = int(key)
if logo_names.has_key(ikey):
full_logo_url = logo_provider[logo_names[ikey][0]]+logo_names[ikey][1]+'.gif'
print ' <icon src="%s" />' % full_logo_url
print ' </channel>'
num_chans = len(channels.keys())
channel_cnt = 0
if program_cache != None:
program_cache.clean()
fluffy = channels.keys()
nfluffy = len(fluffy)
for id in fluffy:
channel_cnt += 1
if not quiet:
sys.stderr.write('\n\nNow fetching %s(xmltvid=%s%s) (channel %s of %s)\n' % \
(channels[id], id, (compat and '.tvgids.nl' or ''), channel_cnt, nfluffy))
info = get_channel_all_days(id, days, quiet)
blah = parse_programs(info, None, quiet)
# fetch descriptions
if not fast:
get_descriptions(blah, program_cache, nocattrans, quiet, slowdays)
# Split titles with colon in it
# Note: this only takes place if all days retrieved are also grabbed with details (slowdays=days)
# otherwise this function might change some titles after a few grabs and thus may result in
# loss of programmed recordings for these programs.
if slowdays == days:
for program in blah:
title_split(program)
print xmlefy_programs(blah, id, desc_len, compat, nocattrans)
# save the cache after each channel fetch
if program_cache != None:
program_cache.dump(program_cache_file)
# be nice to tvgids.nl
time.sleep(random.randint(nice_time[0], nice_time[1]))
if program_cache != None:
program_cache.dump(program_cache_file)
# print footer stuff
print "</tv>"
# close the outputfile if necessary
if output != None:
output.close()
# and return success
sys.exit(0)
# allow this to be a module
if __name__ == '__main__':
main()
# vim:tw=0:et:sw=4
[cedric@tv ~]$
Best regards,
Cedric
Last edited by cdwijs (2010-11-04 18:44:51)

Running the script by python2 solves it for me:
su - mythtv -c "nice -n 19 python2 /usr/bin/tv_grab_nl_py --output ~/listings.xml"
Best regards,
Cedric

Combine Columns From Separate Arrays Into One Formatted Table

What I'm trying to do is make two WMI queries with 2 different classes for a list of machines and then patch the columns together into one single array that is formatted as a table with columns and rows. I seem to keep banging my head against the wall and
I can't help but feel that the answer is simple. I can certainly create an array that contains all 3 columns (such as in the commented out part) but no matter which angle I go at it, it always seems to end up as all the data in one single row in each column
rather than a nicely formatted table. I've even tried constructing separate custom objects and adding the different objects to the array but that's obviously not working. Below is the code of the last thing I tried. I need someone to bash it to death and tell
me the (most likely obvious) thing that I'm doing wrong. Thanks!
$failedos = @()
$failedcs = @()
$ccs = get-adcomputer -property operatingsystem -filter {name -like "*-CC*"} | select name | sort name
$cs = foreach ($cc in $ccs){$cc.name | % {if ($c=get-wmiobject -computername $cc.name -class win32_computersystem -ErrorAction SilentlyContinue){$c | select @{Name="Name";Expression={$_.Name}}, @{Name="Model";Expression={$_.Model}}} else {$failedcs += "$_"}}}
$os = foreach ($cc in $ccs){$cc.name | % {if ($o=get-wmiobject -computername $cc.name -class win32_operatingsystem -ErrorAction SilentlyContinue){$o | select @{Name="OperatingSystem";Expression={$_.caption}}} else {$failedos += "$_"}}}
#[array]$osprops = @{'Name'=$cs.Name;'Model'=$cs.Model;'OperatingSystem'=$os.OperatingSystem}
$result = @()
Foreach ($Line in $cs) {
$MyCustomObject = New-Object -TypeName PSObject
Add-Member -InputObject $MyCustomObject -MemberType NoteProperty -Name "Name" -Value $Line.name -Force
Add-Member -InputObject $MyCustomObject -MemberType NoteProperty -Name "Model" -Value $Line.Model -Force
$result += $MyCustomObject
foreach ($Line2 in $os) {
$MyCustomObject2 = New-Object -TypeName PSObject
Add-Member -InputObject $MyCustomObject2 -MemberType NoteProperty -Name "OperatingSystem" -Value $Line2.OperatingSystem -Force
$result += $MyCustomObject2

Any help?
$ccs = get-adcomputer -property operatingsystem -filter {name -like "*-CC*"} |
select -ExpandProperty name | sort
$Result =
Foreach ($CC in $CCs)
$Object =
New-Object PSObject -Property @{ Name = $CC
Model = 'Failed'
OperatingSystem = 'Failed'
Try {
$Object.Model =
get-wmiobject -computername $CC -class win32_computersystem -ErrorAction Stop |
select -ExpandProperty Model
$Object.OperatingSystem =
get-wmiobject -computername $CC -class win32_operatingsystem -ErrorAction Stop |
select -ExpandProperty Caption
Catch{}
Finally { $Object }
[string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

No data found error on Form on a Table with report

Hi Everyone, I'm using Application Express 4.1.0.00.32 on Windows 7. I built a Form on a table with report. Earlier I was using rowid as a passing parameter but then I had to change it to primary key column from report to form.
So in the "Fetch row process" I changed the "Items containing primary key value" and "Primary Key column" to P1004_PERSON_ID and PERSON_ID respectively. Which is my primary key.
My Form is working exactly fine but at on point it throws "no data found error".
I have a required date field in the form. So if the user doesn't fill in the date field and try to save the form, it throws the "Feild required error" and then when user enters date and try to save then it throws the error "No data found.". here is the snapshot... snapshot
How can I fix this error.I'm really stuck.
I checked debubber..it is as follows... in debughger it's still showing rowid. I don't know why. How can I fix that.
Execution
Message
Level
Graph
0.00233
0.00932
S H O W: application="101" page="1004" workspace="" request="" session="123235901404364"
4
0.01161
0.00102
Language derived from: FLOW_PRIMARY_LANGUAGE, current browser language: en-us
4
0.01261
0.00046
alter session set nls_language="AMERICAN"
4
0.01307
0.00042
alter session set nls_territory="AMERICA"
4
0.01348
0.00053
NLS: CSV charset=WE8MSWIN1252
4
0.01401
0.00042
...NLS: Set Decimal separator="."
4
0.01443
0.00053
...NLS: Set NLS Group separator=","
4
0.01495
0.00050
...NLS: Set g_nls_date_format="DD-MON-RR"
4
0.01545
0.00051
...NLS: Set g_nls_timestamp_format="DD-MON-RR HH.MI.SSXFF AM"
4
0.01597
0.00050
...NLS: Set g_nls_timestamp_tz_format="DD-MON-RR HH.MI.SSXFF AM TZR"
4
0.01647
0.00079
...Setting session time_zone to -05:00
4
0.01726
0.00046
Setting NLS_DATE_FORMAT to application date format: DD-MON-YYYY
4
0.01772
0.00060
Setting NLS_TIMESTAMP_FORMAT to application timestamp format: DD-MON-YYYY HH24.MI.SSXFF
4
0.01832
0.00092
...NLS: Set g_nls_date_format="DD-MON-YYYY"
4
0.01924
0.00049
...NLS: Set g_nls_timestamp_format="DD-MON-YYYY HH24.MI.SSXFF"
4
0.01973
0.00083
...NLS: Set g_nls_timestamp_tz_format="DD-MON-RR HH.MI.SSXFF AM TZR"
4
0.02056
0.00099
NLS: Language=en-us
4
0.02154
0.00157
Application 101, Authentication: PLUGIN, Page Template: 5091946581246503
4
0.02312
0.00065
...fetch session state from database
4
0.02377
0.00106
fetch items
4
0.02483
0.00065
...fetched 103 session state items
4
0.02548
0.00194
Authentication check: NTLM (NATIVE_CUSTOM)
4
0.02742
0.00188
...Execute Statement: begin declare begin wwv_flow.g_boolean := f_ntlm_page_sentry_parm; end; end;
4
0.02930
0.00050
... sentry+verification success
4
0.02980
0.00042
...Session ID 123235901404364 can be used
4
0.03021
0.00114
...Application session: 123235901404364, user=VARMAN01
4
0.03135
0.00162
...Check for session expiration:
4
0.03297
0.00075
Session: Fetch session header information
4
0.03372
0.00113
...Setting session time_zone to -5:00
4
0.03485
0.00080
Branch point: Before Header
4
0.03565
0.00598
Fetch application meta data
4
0.04165
0.00081
...metadata, fetch computations
4
0.04245
0.00076
...metadata, fetch buttons
4
0.04321
0.00086
Setting NLS_DATE_FORMAT to application date format: DD-MON-YYYY
4
0.04406
0.00058
Setting NLS_TIMESTAMP_FORMAT to application timestamp format: DD-MON-YYYY HH24.MI.SSXFF
4
0.04464
0.00049

Just an observance... SQL is still showing the rowid instead of the P1004_PERSON_ID ??
where "PERSON_ID" = :p_rowid;
should it not be :
where "PERSON_ID" = :P1004_PERSON_ID:
thx, Bill

How can I build a table with the time values of a timer from a while loop

Hi:
I have a question concerning building a table:
Every 100ms I read a value from a sensor (while loop with a timer). I would like to build a table with the actual time and the concerning value. For example:
0msec         1V
100msec     2V
200msec     3V
300msec     4V
etc.
If I use the Express VI for building a table, I always get the date and time, but I don't need the date and the time is in the following format: HH:MMS, which is nonsensical for me as I can't differentiate within msec. Can I change the format anywhere?
Can I also save the table to a file or even to an Excelsheet? How can I do that?
Thanks for your help!

Hi Craig:
thank you very much. To solve the mystery : ) :
I want to drive a stepper motor with a specific frequency. To get the current degree value of the motor I would like to measure the current time (from the beginning of the move on). (With a formula I get the degree value out of the time)
Concurrently I would like to get data from a torque sensor and from a pressure sensor. That's why I asked you about the time and the table. The measurement should start with the movement of the motor. How can I do that? Right now I have different block diagrams (different while loops) (see attachment) and I would like to put them in one.
I haven't done the block diagram for the pressure sensor yet, so there is only the one for the torque sensor and the one for the motor.
I also would like to set a mark in the table when the voltage value of an analog input gets under a specific threshold value. Is that possible?
I'm sorry, I'm a novice in LabVIEW. But maybe you can help me.
Thank you very much!
Steffi
Attachments:
motor.vi ‏238 KB
sensor.vi ‏59 KB

Help to read a table with data source and convert time stamp

Hi Gurus,
I have a req and need to write a ABAP prog. As soon as i excute ABAP program it should ask me enter a data source name, then my ABAP prog has excute teh code, in ABAP code i have to read a table with this data source as key, sort time stamp from table and should display the data source and time stamp as output.
As follows:
Enter Data Source Name:
Then user enters : 2lis_11_vahdr
Then out put should be "Data source :" 10-15-2008.
The time stamp format in table is 20,050,126,031,520 (YYYYMMDDhhmmss). I have to display as 05-26-2005. Any help would be apprciated.
Thanks,
Ram

Hi Jayanthi Babu Peruri,
I tried to extract YEAR, MONTH, DAY separately and using
EDIT MASK written it.
Definitely there will be some STANDARD CONVERSION ROUTINE will be there. But no idea about it.
DATA : V_TS      TYPE TIMESTAMP,
       V_TS_T    TYPE CHAR16,
       V_YYYY    TYPE CHAR04,
       V_MM      TYPE CHAR02,
       V_DD      TYPE CHAR02.
START-OF-SELECTION.
GET TIME STAMP FIELD V_TS.
V_TS_T = V_TS.
CONDENSE V_TS_T.
V_YYYY = V_TS_T.
V_MM   = V_TS_T+4(2).
V_DD   = V_TS_T+6(2).
V_TS_T(2) = V_MM.
V_TS_T+2(2) = V_DD.
V_TS_T+4(4) = V_YYYY.
SKIP 10.
WRITE : /10 V_TS," USING EDIT MASK '____-__-________'.
          /10 V_YYYY,
          /10 V_MM,
          /10 V_DD,
          /10 V_TS_T USING EDIT MASK '__-__-__________'.
If you want DATE alone, just declare the length of V_TS_T as 10.
Regards,
R.Nagarajan.
We can -

Uploading a file (.doc, .xls, .txt) into an Oracle table with BLOB column

Hello All :
I have been trying to figure out for a simple code I can use in my JSP to upload a file (of any format) into an Oracle table with a BLOB column type. I have gone through a lot of existing forums but couldnot find a simple code (that doesnot use Servlet, for eg.) to implement this piece.
Thanks a lot for your help !!

Hi.
First of all to put a file into Oracle you need to get the array of bytes byte[] for that file. For this use for example FileInputStream.
After you get the byte array try to use this code:
        try {
            Connection conn = myGetDBConnection();
            PreparedStatement pstmt = conn.prepareStatement("INSERT INTO table1 (content) VALUES(?)");
            byte[] content = myGetFileAsBytes();
            if (content != null) {
                pstmt.setBinaryStream(0, new ByteArrayInputStream(content), content.length);
            pstmt.close();
            conn.close();
        } catch (Exception ex) {
            ex.printStackTrace();
        }or instead of using ByteArrayInputStream try pstmt.setBinaryStream(0, new FileInputStream(yourFile), getFileSize());Hope this will help...
regards, Victor Letunovsky

Tabulator: format tables with Python

Similar Messages

Maybe you are looking for