This is the final post in a 4-part series about configuring and reading logged data using a Kvaser Memorator 2nd Generation device through kvmlib:
Developer Blog
19/09/2018
by Magnus Carlsson
Configure an SD card using kvmlib
2 Adding script and triggers to a configuration
4 Configure an SD card using kvmlib
Earlier we interacted with kvmlib to configure a device and to retrieve both the configuration and logged data for further analysis. Now we will look at what steps that need to be performed differently if we only have access to the SD card that will be inserted into the device. The scenario is where the SD card is shipped between the physical logging location and us.
For the purpose of this exercise, we will take a 16 GB SD card that has been formatted in Windows, initialize and configure this SD card so that we later can insert the SD card into our Kvaser Memorator Pro 2xHS (EAN 00819-9), which is running firmware version 3.11. After running our logging session, we will remove the SD card from the device, read and reset the log files.
4.1 Initialize SD card
If we do not have our device at hand, we can still initialize the SD card using our Windows computer and the mhydraformat.exe program. This will however be more time consuming than using the device, as initializing a 16 GB card takes approximately 3 minutes.
The mhydraformat.exe program can be found in the installation directory of Kvaser Memorator Config Tool.1 You can run the formatting program with the --help argument to see the usage help text.
[C:\]mhydraformat.exe --help
Disk formatting program for Kvaser Memorator (2nd generation) devices.
-h --help | Print this information.
-d=DRIVE --drive=DRIVE | The drive to format, e.g. F:
-r=X --reserve=X | The number of MB to reserve for user files.
-c=X --config=X | The number of MB to reserve for configurations.
--fat16 | Format the disk with FAT16. Default is FAT32.
-i --interactive | Require the user to confirm before format.
-q --quiet | Stop all outputs. Overrides -i.
--bin5 | Use the older, smaller version 5 of PARAM.LIF.
--lio3 | Use the older LIO data format 3 for KMF files.
Example:
mhydraformat -d=F: -r=100 -c=10 --interactiveConnect your SD card to the computer and note the drive letter. For me, the SD card was assigned to J:. We now run mhydraformat.exe to initialize the card. Note that the command line switch to specify the space to be allocated to DATABASE.BIN is here named --config. We also take a look at the contents of the card after the initialization using the Windows dir command.
C:> mhydraformat.exe -d=J: -r=10000 -c=10
Formatting...done.
C:> dir j:
Volume in drive J has no label.
Directory of J:\
2018-08-10 10:39 10 010 624 DATABASE.BIN
2018-08-10 10:39 1 073 741 824 LOG00000.KMF
2018-08-10 10:39 1 073 741 824 LOG00001.KMF
2018-08-10 10:39 1 073 741 824 LOG00002.KMF
2018-08-10 10:39 1 073 741 824 LOG00003.KMF
2018-08-10 10:39 1 073 741 824 LOG00004.KMF
2018-08-10 10:39 514 785 280 LOG00005.KMF
2018-08-10 10:39 10 485 760 PARAM.LIF
8 File(s) 5 903 990 784 bytes
0 Dir(s) 10 004 455 424 bytes freeHere we see the binary configuration file (PARAM.LIF), the log file container (LOG00000.KMF through LOG00005.KMF) and the configuration file reserved for Kvaser Memorator Config Tool (10 MB DATABASE.BIN). We also see 10000 MB of free space as we requested.
4.2 Save configuration to disk
To download the configuration directly to the SD card, we need to open the file system instead of the device as we did in the previous posts. We also need to inform kvmlib about where our SD card was mounted. This is done by calling the function kmfOpen() and supply the full path to the file LOG00000.KMF, which always exists on a correctly initialized SD card.2
We are now also using kvaMemoLibXml to write the binary configuration file directly to the SD card.
As previously stated we also need to supply the correct device type.
# 12_configureSd.py
from canlib import kvamemolibxml
print('kvamemolib version: ', kvamemolibxml.dllversion())
# Our SD card is mounted under J:, so that is where our
# binary configuration should be placed.
filename = 'J:/PARAM.LIF'
# Convert the previously validated XML configuration file
# and write the resulting binary configuration to SD card
kvamemolibxml.kvaXmlToFile("config.xml", filename)Listing 15: Writing binary configuration to SD card.
To download the configuration in clear text, we do just like we did in the previous post, but write the zip file to the SD card directly through the Windows mount point.
# 13_copy_files_to_card.py
import zipfile
# Our SD card is mounted under J:
filename = 'J:/config.zip'
# Creating zip archive
with zipfile.ZipFile(filename, mode='w',
compression=zipfile.ZIP_DEFLATED) as zipf:
# Adding files to zip archive
zipf.write('config.xml')
zipf.write('myCanGenerator.txe')Listing 16: Writing clear text configurations using a zip archive.
If we now look at the contents of our SD card we have the following.
C:>dir J:
Volume in drive J has no label.
Directory of J:\
2018-08-10 11:16 40 604 PARAM.LIF
2018-08-10 10:39 1 073 741 824 LOG00000.KMF
2018-08-10 10:39 1 073 741 824 LOG00001.KMF
2018-08-10 10:39 1 073 741 824 LOG00002.KMF
2018-08-10 10:39 1 073 741 824 LOG00003.KMF
2018-08-10 10:39 1 073 741 824 LOG00004.KMF
2018-08-10 10:39 514 785 280 LOG00005.KMF
2018-08-10 10:39 10 010 624 DATABASE.BIN
2018-08-10 11:21 2 516 config.zip
9 File(s) 5 893 548 144 bytes
0 Dir(s) 10 014 892 032 bytes free4.3 Read result from SD card
When we get the SD card back from the field, we reconnect the SD card to our computer and begin by verifying the LIO data format version on the SD card.
# 14_verifyLioFormatSd.py
from canlib import kvmlib
# Our SD card is mounted under J:, so our LOG00000.KMF can
# be opened from here
filename = 'J:/LOG00000.KMF'
# Open the SD card and read LIO data format version
# We have firmware version 3.0 in the device this SD card will
# be inserted into, this means that the FW is using Lio Data
# Format v5.0 and we should use kvmDEVICE_MHYDRA_EXT as
# the deviceType
with kvmlib.openKmf(
filename,
device_type=kvmlib.Device.MHYDRA_EXT) as kmf:
ldf_version = kmf.log.ldf_version
print('Lio Data Format version:', ldf_version)
# Verify that the LIO data format version of the card corresponds
# to the device type we used when opening the device
if ldf_version != (5, 0):
print('Unexpected Lio Data Format:', ldf_version)
if ldf_version == (3, 0):
print("This log file can be read if you reopen the"
" device as kvmDEVICE_MHYDRA.")Listing 17: Verify LIO data format on SD card.
The next step is to read out the logged data. The only difference from our previous post using a connected device is opening the SD card instead of the device. Go back to the previous post for a more detailed description of what happens here.
# 15_readResultFromSd.py
import glob
import os
from canlib import EAN, VersionNumber
from canlib import kvmlib
# Our SD card is mounted under J:, so our LOG00000.KMF can
# be opened from here
filename = 'J:/LOG00000.KMF'
# Directory to put the resulting files in
resultDir = "result"
# Make sure the result directory exists and is empty
if os.path.isdir(resultDir):
files = glob.glob(os.path.join(resultDir, "*"))
for f in files:
os.remove(f)
else:
os.mkdir(resultDir)
os.chdir(resultDir)
# Open the SD card
# We have earlier verified that the SD card is using Lio Data Format v5.0
# and we should use kvmDEVICE_MHYDRA_EXT as the deviceType
with kvmlib.openKmf(filename) as kmf:
ldf_version = kmf.log.ldf_version
# Verify that the LIO data format of the card corresponds to
# the device type we used when opening the device
if ldf_version != VersionNumber(major=5, minor=0):
if ldf_version == VersionNumber(major=3, minor=0):
raise ValueError('This log file can be read if you reopen the'
' device as kvmDEVICE_MHYDRA.')
raise ValueError('Unexpected Lio Data Format: ' + str(ldf_version))
# Read number of recorded logfiles
num_log_files = len(kmf.log)
print("Found {num} file on card: ".format(num=num_log_files))
# Loop through all logfiles and write their contents to .kme50 files
for i, log_file in enumerate(kmf.log):
print("\n==== File {index}: {start} - {end}, approx {num} events".
format(index=i,
start=log_file.start_time,
end=log_file.end_time,
num=log_file.event_count_estimation()))
# The first logEvent contains device information
event_iterator = iter(log_file)
first_event = next(event_iterator)
ean = EAN.from_hilo([first_event.eanHi, first_event.eanLo])
serial = first_event.serialNumber
# Add EAN and serial number info to filename
logfile_name = ('log_{ean}_{sn}_{index:03}.kme50'.
format(ean=str(ean), sn=serial, index=i))
print('Saving to:', logfile_name)
with kvmlib.createKme(logfile_name) as kme:
print(first_event)
kme.write_event(first_event)
for event in event_iterator:
# Write event to stdout
print(event)
kme.write_event(event)
# Delete all logfiles
# kmf.log.delete_all()Listing 18: Read logged data from SD card and save to .kme50 files.
Running the above code we can see that six messages from each channel turns up inside the log.
Found 1 file on card:
==== File 0: 2018-08-10 12:26:38 - 2018-08-10 12:26:44, approx 20 events
Saving to: log_73-30130-00819-9_11573_000.kme50
*t: - EAN:73-30130-00819-9 s/n:11573 FW:v3.11.557 LIO:v5.0
t: 2.701738275 DateTime: 2018-08-10 12:26:38
t: 2.701738275 Log Trigger Event (type: 0x2, trigno: 0x01, pre-trigger: 0, post-trigger: 0)
t: 2.701738275 ch:1 f: 2 id: 3 dlc: 8 d:12 21 13 31 22 34 41 15
t: 2.701738337 Log Trigger Event (type: 0x2, trigno: 0x01, pre-trigger: 0, post-trigger: 0)
t: 2.701738337 ch:0 f: 42 id: 3 dlc: 8 d:12 21 13 31 22 34 41 15
t: 3.7017584 ch:1 f: 2 id: 4 dlc: 8 d:12 21 13 31 22 34 41 15
t: 3.701758462 ch:0 f: 42 id: 4 dlc: 8 d:12 21 13 31 22 34 41 15
t: 4.701774525 ch:1 f: 2 id: 5 dlc: 8 d:12 21 13 31 22 34 41 15
t: 4.701774587 ch:0 f: 42 id: 5 dlc: 8 d:12 21 13 31 22 34 41 15
t: 5.70179165 Log Trigger Event (type: 0x2, trigno: 0x01, pre-trigger: 0, post-trigger: 2500)
t: 5.70179165 ch:1 f: 2 id: 6 dlc: 8 d:12 21 13 31 22 34 41 15
t: 5.701791712 ch:0 f: 42 id: 6 dlc: 8 d:12 21 13 31 22 34 41 15
t: 6.701809775 ch:1 f: 2 id: 7 dlc: 8 d:12 21 13 31 22 34 41 15
t: 6.701809837 ch:0 f: 42 id: 7 dlc: 8 d:12 21 13 31 22 34 41 15
t: 7.7018359 ch:1 f: 2 id: 8 dlc: 8 d:12 21 13 31 22 34 41 15
t: 7.701835962 ch:0 f: 42 id: 8 dlc: 8 d:12 21 13 31 22 34 41 154.4 Read configuration from SD card
For completeness, the only thing left is to show how to read back our compressed configuration that we placed on the SD card earlier using the Windows mount point.
# 16_readConfigFromSd.py
import glob
import os
import shutil
# Our SD card is mounted under J:, so our LOG00000.KMF can
# be opened from here
filename = 'J:/LOG00000.KMF'
for file in glob.glob(os.path.join(os.path.dirname(filename), "*.*")):
if(os.path.splitext(file)[1].lower() == '.kmf'
or file.lower() == 'param.lif'
or file.lower() == 'database.bin'):
print('Skipping:', file)
else:
print('Copying:', file)
shutil.copy(file, ".")Listing 19: Copy of all user files from the SD card.
Now everything from this run is placed inside the result directory.
C:> dir result
Directory of result
2018-08-10 13:28 2 516 config.zip
2018-08-10 13:28 10 010 624 DATABASE.BIN
2018-08-10 13:16 452 log_73-30130-00819-9_11573_000.kme50
2018-08-10 13:28 40 604 PARAM.LIF
4 File(s) 10 054 196 bytes
2 Dir(s) 786 525 990 912 bytes freeThis was all for now, hopefully this blog series has helped you getting to know more about kvmlib and showed what the library can do for you when managing Kvaser logger devices.
Footnotes
Original Article
Original article published April 19, 2016.
This is the final post in a 4-part series about configuring and reading logged data using a Kvaser Memorator 2nd Generation device through kvmlib:
2 Adding script and triggers to a configuration
4 Configure an SD card using kvmlib
Earlier we interacted with kvmlib to configure a device and to retrieve both the configuration and logged data for further analysis. Now we will look at what steps that need to be performed differently if we only have access to the SD card that will be inserted into the device. The scenario is where the SD card is shipped between the physical logging location and us.
For the purpose of this exercise, we will take a 16 GB SD card that has been formatted in Windows, initialize and configure this SD card so that we later can insert the SD card into our Kvaser Memorator Pro 5xHS (EAN 00778-9), which is running firmware version 3.0. After running our logging session, we will remove the SD card from the device, read and reset the log files. Full program listings are available on GitHub.
Update 2017-03-21: Updated sample code to work with Python 3 as well as using the Python canlib package introduced in CANlib SDK v5.17.
Update 2017-03-28: Added the now preferred usage of kmfWriteConfig() over letting kvaXmlToFile() write directly to the SD Card.
4.1 Initialize SD card
If you do not have your device at hand, you can still initialize the SD card using your computer and the mhydraformat.exe program. This will however be more time consuming than using the device, as initializing a 16 GB card takes approximately 3 minutes.
The mhydraformat.exe program can be found in the installation directory of Kvaser Memorator Config Tool.1 You can run the formatting program with the –help argument to see the usage help text.
[C:\]mhydraformat.exe --help Disk formatting program for Kvaser Memorator (2nd generation) devices.
-h    --help | Print this information.
-d=DRIVE --drive=DRIVE | The drive to format, e.g. F:
-r=X Â Â --reserve=X | The number of MB to reserve for user files.
-c=X Â Â --config=X Â Â | The number of MB to reserve for configurations.
--fat16 | Format the disk with FAT16. Default is FAT32.
-i --interactive | Require the user to confirm before format.
-q --quiet | Stop all outputs. Overrides -i.
--bin5 | Use the older, smaller version 5 of PARAM.LIF.
--lio3 | Use the older LIO data format 3 for KMF files.
Example:
mhydraformat -d=F: -r=100 -c=10 --interactive
Connect your SD card to the computer and note the drive letter. For me, the SD card was assigned to E:. After inspecting the contents of the SD card, we run mhydraformat.exe to initialize the card. Note that the command line switch to specify the space to be allocated to DATABASE.BIN is here named –config. We also take a look at the contents of the card after the initialization using the Windows dir command.
dir E:
Volume in drive E has no label.
Volume Serial Number is 5C92-4738
Directory of E:\
2016-02-23 08:41 0 dummy_file
1 File(s) 0 bytes
0 Dir(s) 16Â 129Â 335Â 296 bytes free
mhydraformat.exe -d=E: -r=10000 -c=10
Formatting...done.
dir E:
Volume in drive E has no label.
Directory of E:\
2016-02-23 08:45 10Â 485Â 760 PARAM.LIF
2016-02-23 08:45 1Â 073Â 741Â 824 LOG00000.KMF
2016-02-23 08:45 1Â 073Â 741Â 824 LOG00001.KMF
2016-02-23 08:45 1Â 073Â 741Â 824 LOG00002.KMF
2016-02-23 08:45 1Â 073Â 741Â 824 LOG00003.KMF
2016-02-23 08:45 1Â 073Â 741Â 824 LOG00004.KMF
2016-02-23 08:45 736Â 649Â 216 LOG00005.KMF
2016-02-23 08:45 10Â 010Â 624 DATABASE.BIN
8 File(s) 6Â 125Â 854Â 720 bytes
0 Dir(s) 10Â 004Â 480Â 000 bytes free
echo
ECHO is on.
Here we see the binary configuration file (PARAM.LIF), the log file container (LOG00000.KMF through LOG00005.KMF) and the configuration file reserved for Kvaser Memorator Config Tool (10 MB DATABASE.BIN). We also see 10000 MB of free space as we requested.
4.2 Save configuration to disk
To download the configuration directly to the SD card, we need to open the file system instead of the device as we did in one of the previous posts. We also need to inform kvmlib about where our SD card was mounted. This is done by calling the function kmfOpen() and supply the full path to the file LOG00000.KMF, which always exists on a correctly initialized SD card.2
We are now also using kvaMemoLibXml to write the binary configuration file directly to the SD card.
As previously stated we also need to supply the correct device type.
import canlib.kvaMemoLibXml as kvaMemoLibXml
import canlib.kvmlib as kvmlib
# Our SD card is mounted under E:, so that is where our
# binary configuration should be placed.
kmfFilename = "E:\\LOG00000.KMF"
ml = kvmlib.kvmlib()
xl = kvaMemoLibXml.kvaMemoLibXml()
print("kvaMemoLibXml version: v%s" % xl.getVersion())
# Convert the previously validated XML configuration file
# and write the resulting binary configuration to SD card
# It is no longer recommended to write directly to the SD Card
# using kvaXmlToFile() but use the kmfWriteConfig()
# function instead.
with open("config.xml", 'r') as myfile:
configXml = myfile.read()
try:
# Convert the previously validated XML configuration file
configBuf = xl.kvaXmlToBuffer(configXml)
except kvaMemoLibXml.kvaError:
print("Error in configuration.")
exit(1)
try:
# Write the resulting binary configuration to SD card
deviceType = kvmlib.kvmDEVICE_MHYDRA_EXT
ml.kmfOpen(filename=kmfFilename, deviceType=deviceType)
ml.kmfWriteConfig(configBuf)
ml.close
except kvmlib.kvmDiskNotFormated:
print("SD card is not initialized.")
exit(1)
Listing 18: Writing binary configuration to SD card.
To download the configuration in clear text, we do just like we did in the previous post, but write the zip file to the SD card directly through the Windows mount point.
import zipfile
# Our SD card is mounted under E:
filename = "E:\\config.zip"
# Creating zip archive
with zipfile.ZipFile(filename, mode='w',
compression=zipfile.ZIP_DEFLATED) as zipf:
# Adding files to zip archive
zipf.write("config.xml")
zipf.write("myCanGenerator.txe")Listing 19: Writing clear text configurations using a zip archive.
If we now look at the contents of our SD card we have the following.
d:\temp\blog>dir E:
Volume in drive E has no label.
Directory of E:\
2016-02-19 07:58 40Â 308 PARAM.LIF
2016-02-11 14:43 1Â 073Â 741Â 824 LOG00000.KMF
2016-02-11 14:43 1Â 073Â 741Â 824 LOG00001.KMF
2016-02-11 14:43 1Â 073Â 741Â 824 LOG00002.KMF
2016-02-11 14:43 1Â 073Â 741Â 824 LOG00003.KMF
2016-02-11 14:43 1Â 073Â 741Â 824 LOG00004.KMF
2016-02-11 14:43 744Â 030Â 208 LOG00005.KMF
2016-02-11 14:43 10Â 002Â 432 DATABASE.BIN
2016-02-19 10:10 2Â 406 config.zip
9 File(s) 6Â 122Â 784Â 474 bytes
0 Dir(s) 10Â 010Â 435Â 584 bytes free
4.3 Read result from SD card
When we get the SD card back from the field, we reconnect the SD card to our computer and begin by verifying the LIO data format version on the SD card.
import canlib.kvmlib as kvmlib
ml = kvmlib.kvmlib()
# Our SD card is mounted under E:, so our LOG00000.KMF can
# be opened from here
filename = "E:\\LOG00000.KMF"
try:
# Open the SD card
# We have firmware version 3.0 in the device this SD card will
# be inserted into, this means that the FW is using Lio Data
# Format v5.0 and we should use kvmDEVICE_MHYDRA_EXT as
# the deviceType
lioDataFormat = ml.kmfOpenEx(filename,
deviceType=kvmlib.kvmDEVICE_MHYDRA_EXT)
print("Lio Data Format v%s" % lioDataFormat)
# Close SD card
ml.close()
# Verify that the LIO data format of the card corresponds to
# the device type we used when opening the device
if lioDataFormat != '5.0':
print("Unexpected Lio Data Format:", lioDataFormat)
if lioDataFormat == '3.0':
print("This log file can be read if you reopen the"
" device as kvmDEVICE_MHYDRA.")
exit(1)
except kvmlib.kvmDiskNotFormated:
print("SD card is not initialized...")
exit(1)Listing 20: Verify LIO data format on SD card.
The next step is to read out the logged data. The only difference from our previous post using a connected device is opening the SD card instead of the device. Go back to the previous post for a more detailed description of what happens here.
import glob
import os
import canlib.kvmlib as kvmlib
ml = kvmlib.kvmlib()
# Our SD card is mounted under E:, so our LOG00000.KMF can
# be opened from here
filename = "E:\\LOG00000.KMF"
# Directory to put the resulting files in
resultDir = "result"
# Make sure the result directory exists and is empty
if os.path.isdir(resultDir):
files = glob.glob(os.path.join(resultDir, "*"))
for f in files:
os.remove(f)
else:
os.mkdir(resultDir)
os.chdir(resultDir)
# Open the SD card
# We have earlier verified that the SD card is using Lio Data Format v5.0
# and we should use kvmDEVICE_MHYDRA_EXT as the deviceType
ml.kmfOpen(filename, deviceType=kvmlib.kvmDEVICE_MHYDRA_EXT)
# Read number of recorded logfiles
fileCount = ml.logFileGetCount()
print("Found %d file%s on card:" % (fileCount,
"s" if fileCount > 1 else ""))
# Loop through all logfiles and write their contents to .kme50 files
for fileIndx in range(fileCount):
eventCount = ml.logFileMount(fileIndx)
print("\tFile %3d: %10d events" % (fileIndx, eventCount))
logEvent = ml.logFileReadEventLogFormat()
# The first logEvent contains device information
memoEvent = logEvent.createMemoEvent()
sn = memoEvent.serialNumber
ean_lo = memoEvent.eanLo
ean_sn = "%05x-%x_%d" % ((ean_lo >> 4) & 0xfffff, ean_lo & 0xf, sn)
# Add EAN and serial number info to filename
logfileName = "log_%s_%d.kme50" % (ean_sn, fileIndx)
ml.kmeCreateFile(logfileName, kvmlib.kvmFILE_KME50)
while logEvent is not None:
# Write event to stdout
print(logEvent)
ml.kmeWriteEvent(logEvent)
# Read next event
logEvent = ml.logFileReadEventLogFormat()
ml.kmeCloseFile()
# Delete all logfiles
ml.logFileDeleteAll()
# Close device
ml.close()Listing 21: Read logged data from SD card and save to .kme50 files.
4.4 Read configuration from SD card
For completeness, the only thing left is to show how to read back our compressed configuration that we placed on the SD card earlier using the Windows mount point.
import glob
import os
import shutil
# Our SD card is mounted under E:, so our LOG00000.KMF can
# be opened from here
filename = "E:\\LOG00000.KMF"
for file in glob.glob(os.path.join(os.path.dirname(filename), "*")):
if(os.path.splitext(file)[1].lower() == '.kmf'
or file.lower() == 'param.lif'
or file.lower() == 'database.bin'):
print("Skipping %s" % file)
else:
print("Copying %s" % file)
shutil.copy(file, ".")Listing 22: Copy of all user files from the SD card.
Copying E:\\PARAM.LIF Skipping E:\\LOG00000.KMF Skipping E:\\LOG00001.KMF Skipping E:\\LOG00002.KMF Skipping E:\\LOG00003.KMF Skipping E:\\LOG00004.KMF Skipping E:\\LOG00005.KMF Copying E:\\DATABASE.BIN Copying E:\\config.zip
Now everything from this run is placed inside the result directory.
d:\temp\blog>dir result
Volume in drive D is HDD
Volume Serial Number is 26E3-B474
Directory of d:\temp\blog\result
2016-02-22 16:06 <DIR> .
2016-02-22 16:06 <DIR> ..
2016-02-22 16:06 2Â 406 CONFIG.ZIP
2016-02-22 16:06 10Â 002Â 432 DATABASE.BIN
2016-02-22 16:06 40Â 308 PARAM.LIF
3 File(s) 10Â 045Â 146 bytes
2 Dir(s) 961Â 850Â 183Â 680 bytes free
d:\temp\blog>More?
This was all for now, hopefully this blog series has helped you getting to know more about kvmlib and showed what the library can do for you when managing Kvaser logger devices.
Footnotes
