Compression on-calc: LZFO1

RSS
Email

0 Kudos

Author	Comment
bremen	Compression on-calc: LZFO1	Lead	[-]
Posts: 91 (2003-10-02 17:47:12) Reply Quote More My Recent Posts	I have uploaded a first beta of my new program to compress on-calc, you'll find it at utenti.lycos.it/bremen79/ftp/ as lzfo1.zip. The features are: - high compression ratio (similar to PepZip) - very fast decompression (faster than you can imagine! ) - compression almost 2 times faster than PepZip I hope you'll test it! Please report any bugs or errors, suggestions, etc. Bad news: I have not yet included the source of the compression library because I'm thinking to write a paper on it, I must first talk with some professor of my university. I don't know when, but, however, I'll release it under GPL license.
Kevin Kofler	Re: Compression on-calc: LZFO1	#1	[-]
Posts: 7071 (2003-10-02 19:53:47) Admin Reply Quote More My Recent Posts	*Quote:* - high compression ratio (similar to PepZip) Just curious: How does it compare to PC-based compressors such as ttpack? But I guess I should just test it with a few files myself. *Quote:* I have not yet included the source of the compression library because I'm thinking to write a paper on it, I must first talk with some professor of my university. Wow, you actually have the courage to write a paper on some technique you wrote in your spare time... Most hobby programmers (including me, even though I do study Computer Science, along with Mathematics) just release everything, and are way too lazy to write papers. I'll wish you good luck with your paper. In bocca al lupo! Let's hope it gets published in something Computer Science people read. Maybe you'll even become famous.
Fisch2	Re: Compression on-calc: LZFO1	#2	[-]
Posts: 548 (2003-10-02 21:20:04) Reply Quote More My Recent Posts	On the first file that I tried to compress, it said there was an error compressing and then the calc reset . I used: packtest("VarName"). The file was already compressed using ttpack so that could be the problem but the program should still not cause a reset.
bremen	Re: Compression on-calc: LZFO1	#3	[-]
Posts: 91 (2003-10-02 21:41:53) Reply Quote More My Recent Posts	Kevin: *Quote:* Just curious: How does it compare to PC-based compressors such as ttpack? It is inferior for compression ratio, but ttpack uses an optimal parsing strategy: it is a very slow method! Moreover the most part of the PC compressor uses an LZ77 stage plus an huffman/fano/shannon/arithmetic coder, like lzh (PepZip), that is they are slower. *Quote:* Most hobby programmers (including me, even though I do study Computer Science, along with Mathematics) just release everything, and are way too lazy to write papers. I'll wish you good luck with your paper. In bocca al lupo! In the next month I will start a PhD in EE and informatics, so... It is a good start! Thanks for the wishes! Crepi! Fisch2: *Quote:* I used: packtest(... I think you have downloaded the wrong file! 'packtest' is a file of the old packlib. However, what version is? 0.50 or 0.70? The new compressor is in the lzfo1.zip.
bremen	Re: Compression on-calc: LZFO1	#4	[-]
Posts: 91 (2003-10-03 18:30:10) Reply Quote More My Recent Posts	New beta (0.53) uploaded: - optimizations in the compression and decompression functions: compression 0.7% faster and program -228 bytes - Fixed two bugs that could happen in very rare cases during compression The link is the same.
Kevin Kofler	Re: Compression on-calc: LZFO1	#5	[-]
Posts: 7071 (2003-10-03 22:10:19) Admin Reply Quote More My Recent Posts	I updated my old ttpack/shrnklib/pk92lib comparison to include on-calc compressors, and here are the results: Test files:* text: CHEMISLV 4.10 English help file* exec: Phoenix 7.3 TI-89 executable* pict: upper 77 rows of TI-89 XtraKeys 2.16 keymap in 89i formatTested packers:bin: uncompressed file size after ttstripp92: Packer92 (PC-based compression, on-calc decompression)shr: Shrink92 (PC-based compression, on-calc decompression)ttp: ttpack (PC-based compression, on-calc decompression)zip: ZipLib (on-calc compression and decompression)pep: PepZip 1.11 (on-calc compression and decompression)lzf: LZFO1 (on-calc compression and decompression)Results: \| text ratio \| exec ratio \| pict ratio \| total ratio \| rankbin \| 5794 -0% \| 15897 -0% \| 1547 -0% \| 23238 -0% \| 7p92 \| 2974 -49% \| 10896 -31% \| 992 -36% \| 14862 -36% \| 5shr \| 2892 -50% \| 10414 -34% \| 1008 -35% \| 14314 -38% \| 4ttp \| 2913 -50% \| 10019 -37% \| 888 -43% \| 13820 -41% \| 2zip \| 3936 -32% \| 12678 -20% \| 1084 -30% \| 17698 -24% \| 6pep \| 2907 -50% \| 9931 -38% \| 875 -43% \| 13713 -41% \| 1lzf \| 3120 -46% \| 10088 -37% \| 893 -42% \| 14101 -39% \| 3 As you can see, LZFO1 is third (it beats shrnklib and pk92lib). But the biggest surprise for me is that PepZip comes out first, beating ttpack on all 3 files, and beaten only on the text file by shrnklib (which wins that specific test) over 15 bytes. Not a big surprise is that ziplib loses completely. Only leaving the files completely uncompressed gives you a worse ratio than ziplib, and that on all 3 files. [EDIT: Fixed minor errors (LZFO->LZFO1, and it is 15 bytes, not 9).]
Fisch2	He-he	#6	[-]
Posts: 548 (2003-10-04 00:07:16) Reply Quote More My Recent Posts	>>what version is? 0.50 or 0.70? >>The new compressor is in the lzfo1.zip. Whoops! I'm pretty sure the one I downloaded was 0.70, which included three separate compression executables for the 89 bin.
Samuel Stearley	reply to thy	#7	[-]
Posts: 1211 (2003-10-04 01:34:44) Reply Quote More My Recent Posts	Kevin I ran the entire 92+ help picture of extra keys through image studio, RLE compression compressed it 59% of origional. For the ti-89 pic horizontal RLE compressed it to 86% and vertical rle compressed it to 83%
Lionel Debroux	Re: reply to thy	#8	[-]
Posts: 3079 (2003-10-04 13:23:07) Former Admin Reply Quote More My Recent Posts	Great work Francesco ! The compression time is acceptable (148 ticks to compress tictexpl, 27541 bytes, to 17097 bytes); the decompression time is very low (only 14 ticks to decompress tictexpl !); the compression ratios are good (much better than other on-calc compressions, except Pepzip of course, but LZFO is not so far from Pepzip). I think supporting it in ttstart (decompression) and tictexpl (both compression and decompression) would be interesting. But I need help to integrate it (I wouldn't be able to do it quickly). Is anyone willing to / has enough time to help me ? If we could have the code, we might tweak a bit for speed on M68k. For example, TIGCC doesn't always generate good code when working with arrays of elements whose size is not 1 byte (since it can't know the actual size of the array, it uses the always-safe way, under-optimized in that case). Macros or use of predecremented and postincrementing addressing modes can be used to improve the code in most cases. Compare HeapDeref on AMS 2.xx (always-safe way) and on AMS 1.xx (optimized way, works only with arrays whose size is = 32768 bytes), GCC in TIGCC does exactly the same thing as the compiler in TIFS. On AMS 1.xx, HeapDeref is probably written using a macro; on AMS 2.xx, it clearly isn't.
bremen	Re: New version	#9	[-]
Posts: 91 (2003-10-04 15:19:54) Reply Quote More My Recent Posts	I have uploaded a new version (0.54), I have fixed a stupid behaviour of the lazy matching method, now compression ratio is slightly better. Kevin can send me the files you have used for your comparison? However it is always difficult to compare different compressors; on the PC version of my program I am using the Calgary Corpus and Canterbury Corpus tests. We should define a set of "typical" files even for the calculator, for example 2 files for each category (asm, basic, pic, text, ecc.) In my tests I use 8 different files and ttpack is the better on 7! Fisch2: I will test the Complib 0.70 with packed files, however can you send me the compressed file you have used? Lionel: now tictexpl is compressed to 17088 while PepZip 2.00 compress the file only to 17107 Supporting LZFO1 in tictexpl can be a good idea, while I don't think that can be useful in ttstart: ttpack is (yet?) the best for asm file. Moreover LZFO1 require 32772 bytes of memory in decompression, while ttpack doesn't require additional memory. When I'll release the source I will be happy to follow your suggestions on the code optimizations, be patient!
Lionel Debroux	Re: New version	#10	[-]
Posts: 3079 (2003-10-05 14:31:50) Former Admin Reply Quote More My Recent Posts	> Supporting LZFO1 in tictexpl can be a good idea, while I don't think that can be useful in ttstart: ttpack is (yet?) the best for asm file. > Moreover LZFO1 require 32772 bytes of memory in decompression, while ttpack doesn't require additional memory. Ah, yes, you're right, I didn't think about that. However, that's not usually a big problem when the major part of the RAM is free. Indeed, a compressed program is compressed because it is large, so it needs much RAM to run. And the RAM required by the decompression is not used anymore when the decompression is finished. > When I'll release the source I will be happy to follow your suggestions on the code optimizations, be patient! Hmm, I want to be sure you didn't take me wrong. I wasn't requesting you to give us the sourcecode so that we could tweak it a bit for speed ! Of course I'll wait for it to be released. If we want an optimal bench, I think we should not only bench two files of the same kind, but more (10 ?), especially for ASM programs, texts and general data (external data files) which are the types most frequently compressed.
bremen	Re: New version 0.60 beta	#11	[-]
Posts: 91 (2003-10-07 18:46:40) Reply Quote More My Recent Posts	New version (0.60): - Fixed an important bug in the use of the working memory in compression - Various optimizations in compression and decompression functions - Improved the lazy matching method: compression ratio increased of 0.7% , 8% slower in compression For example now tictexpl is compressed to 16996, 111 bytes less than PepZip! Please test it! Lionel: *Quote:* However, that's not usually a big problem when the major part of the RAM is free. Indeed, a compressed program is compressed because it is large, so it needs much RAM to run. You are right! However if you want to integrate LZFO1 in tictexpl I can help you. *Quote:* If we want an optimal bench, I think we should not only bench two files of the same kind, but more (10 ?), especially for ASM programs, texts and general data (external data files) which are the types most frequently compressed. I'll try to create a standard corpus for the test, then I'll post it.
bremen	Re: New version 0.64 beta	#12	[-]
Posts: 91 (2003-10-13 16:33:22) Reply Quote More My Recent Posts	Now compression is 2.1% faster and the program is smaller. I have also created a first draft of a standard corpus for compression tests. Check the html in the archive (lzfo1.zip in utenti.lycos.it/bremen79/ftp/) for the results of the tests on LZFO1, PepZip and TTPack. The summary is that LZFO1 has a better global compression ratio than PepZip, while has a lesser mean compression ratio. That is the compression is better than PepZip only on medium and big files. TTPack has the best mean and global compression ratio. Moreover I'm working on a new version that uses only 16Kb in decompression (now LZFO1 uses 32Kb) and has a better compression ratio! Any suggestion, bug report, simple opinion is welcome!
Lionel Debroux	Re: New version 0.64 beta	#13	[-]
Posts: 3079 (2003-10-13 16:49:18) Former Admin Reply Quote More My Recent Posts	> Now compression is 2.1% faster and the program is smaller. No change in the compression ratio, I guess ? That said, having smaller and faster compression routines is already very good. > Moreover I'm working on a new version that uses only 16Kb in decompression (now LZFO1 uses 32Kb) and has a better compression ratio! Great ! > simple opinion is welcome! You have mine :D
Lionel Debroux	Re: New version 0.64 beta	#14	[-]
Posts: 3079 (2003-12-25 19:41:58) Former Admin Reply Quote More My Recent Posts	I've seen Pollux released XPak. Does anyone have time to compare LZFO1, Pepzip, TTPack, XPak ?
bremen	New version 0.65 beta + XPak comparison	#15	[-]
Posts: 91 (2003-12-26 20:52:02) Reply Quote More My Recent Posts	I have made another version of LZFO1, I have disabled the auto-int 1 and now compression is 12% faster and decompression is 15% faster! I have also added XPak to the comparison table, however in my tests it is slower than lzfo1 in compression and it crashes during decompression. The new file is at utenti.lycos.it/bremen79/ftp/
Lionel Debroux	Re: New version 0.65 beta + XPak comparison	#16	[-]
Posts: 3079 (2003-12-28 16:23:27) Former Admin Reply Quote More My Recent Posts	Great ! I advise you to release LZFO1 quickly on ticalc and have ticalc feature it (since it is more powerful than XPak which was featured). Not that I hate Pollux (anyway he's an excellent programmer with more skills than I have - but he's often too greedy, even if he's releasing more and more interesting programs), but that's because your compression seems to be more powerful than his for most data types (especially ASM). I'm always willing to integrate the LZFO1 compression/decompression in TICT-Explorer. Adding the XPak compression/decompression as well will certainly necessary (both compressions are bound to be used in the mid-term), but if Pollux doesn't provide at least an archive and a header, that won't be possible... Including both compressions will increase the size importantly (10KB+ ? over currently 22 KB, but I don't think I can optimize much more). Note: did you test XPak with the compression which compresses text better (I think I read somewhere in the docs that such compression exists) ?
bremen	Re: New version 0.65 beta + XPak comparison	#17	[-]
Posts: 91 (2003-12-29 18:17:19) Reply Quote More My Recent Posts	Lionel I have updated the comparison chart using the "text compression" of XPak when better. I have also decreased the amount of memory required for the decompression of lzfo1: now it is 2*size of the packed file if it is < 16384, otherwise 32768. Download the new version (0.66) at the usual place. BTW, Lionel thanks for the encouragements!
Pollux	Re: New version 0.65 beta + XPak comparison	#18	[-]
Posts: 115 (2004-01-12 04:46:58) Reply Quote More My Recent Posts	*Quote:* I have also added XPak to the comparison table, however in my tests it is slower than lzfo1 in compression That's possible, XPak is optimized for decompression speed rather than compression speed (like TTPack, actually) BTW, your comparison is a very good idea, here are a few suggestions : - you could also add compression/decompression times (I saw a program called "ticktest" that automatically measures how long a program takes to run, to make it easier for you) - you could add partial summaries so you can figure out which compression is best for which purposes - another idea would be to add shrnklib (I was told it performed quite well on graphics stuff), and -- just to put it all in perspective ;-) -- ziplib... *Quote:* and it crashes during decompression. Some people have already reported this kind of bug, which seem to be related to one of the .89g in my zipfile being corrupted. I'll try to have a look at it when I have enough time. It appears to work fine if you manage to get the proper file on your calculator, though. Lionel> *Quote:* but he's often too greedy I'm truly wondering what you're meaning here. Am I making money off all this? *Quote:* your compression seems to be more powerful than his for most data types (especially ASM). Hum, actually, for ASM programs, GTC features a modified version of the XPak compression that works much better on ASM programs (especially those generated with GTC, since it can generate code that's slightly tuned to optimize redundancy). *Quote:* Adding the XPak compression/decompression as well will certainly necessary (both compressions are bound to be used in the mid-term), but if Pollux doesn't provide at least an archive and a header, that won't be possible... GTC comes in standard with XPak decompression (as well as ttunpack), but I didn't create a TIGCC archive yet (not enough time...) FYI, Tict Explorer compiles just fine under GTC with a straightforward patch for assembly statements, so as of now you may enable XPak decompression under GTC, and afterwards enable it under TIGCC too when I manage to create the TIGCC headers. (in addition, GTC would allow to tictexpl to be way smaller thanks to its self-packer, at the expense of a small loading time [a fraction of a second] every time you run a program) *Quote:* Including both compressions will increase the size importantly (10KB+ ? over currently 22 KB, but I don't think I can optimize much more). I think compressions would be better suited for a "library" environment, as they are often rather large (5 kb for XPak). But you could still include all the decompressors (XPak's is only 1 kb). [EDIT by Kevin Kofler: fixed the last 2 quote tag pairs: "cite" -> "quote".] [EDIT by Pollux : precised somehow vague formulation]
Lionel Debroux	Re: New version 0.65 beta + XPak comparison	#19	[-]
Posts: 3079 (2004-01-12 18:45:34) Former Admin Reply Quote More My Recent Posts	> BTW, your comparison is a very good idea, here are a few suggestions: > - you could also add compression/decompression times (I saw a program called "ticktest" that automatically measures how long a program takes to run, to make it easier for you) There's also program "timing" by Bhuvanesh. > - you could add partial summaries so you can figure out which compression is best for which purposes Hmm, that can't hurt, but don't you think readers should be able to see that by themselves ? > - another idea would be to add shrnklib (I was told it performed quite well on graphics stuff), and -- just to put it all in perspective ;-) -- ziplib... This is useless in my opinion, since ttpack / Pepzip / LZFO1 / XPak are based on different and more efficient algorithms, every programmer should know it. > Quote: but he's often too greedy > I'm truly wondering what you're meaning here. Am I making money off all this? And I'm really wondering if you really don't know what the others think about you... You know the entire TI community knows you as "Mister Vaporware" (OK, this is less true than it used to be, since you released several programs), don't you ? > Quote: your compression seems to be more powerful than his for most data types (especially ASM). > Hum, actually, for ASM programs, GTC features a modified version of the XPak compression that works much better on ASM programs (especially those generated with GTC, since it can generate code that's slightly tuned to optimize redundancy). Do you mean it's incompatible with the standard XPak ? If you do, don't expect me to support both. I don't even know if I'll support any (even in decompression), due to the way you behave (see below)... > Quote: Adding the XPak compression/decompression as well will certainly necessary (both compressions are bound to be used in the mid-term), but if Pollux doesn't provide at least an archive and a header, that won't be possible... > GTC comes in standard with XPak decompression (as well as ttunpack) I know, it's written above, but I don't really care about your non-standards-compliant compiler which will therefore in my opinion bring more incompatibilities and bad things in the community. The only advantage of GTC is that it will allow on-calc programming. > but I didn't create a TIGCC archive yet (not enough time...) Ah, yes, I forgot that creating a TIGCC archive was very very hard and time-consuming... Are you laughing at us ? > FYI, Tict Explorer compiles just fine under GTC I don't care, and this is possibly no longer true anyway: the optimizations I made to reduce its size include several blocks of inline ASM with C operands (unreleased beta version). > so as for now you may enable XPak decompression under GTC, and afterwards enable it under TIGCC too when I manage to create the TIGCC headers. N/A. > (in addition, GTC would allow to tictexpl to be way smaller thanks to its self-packer, at the expense of a small loading time [a fraction of a second] every time you run a program) The ttpack compression is probably more powerful and fast enough (decompression rate is >30 KB/s, timed approximately with a watch), but overall, it is much more used (all file explorers and PedroM support it)... Anyway, your compiler is non-standards-compliant. > Quote: Including both compressions will increase the size importantly (10KB+? over currently 22 KB, but I don't think I can optimize much more). > I think compressions would be better suited for a "library" environment, as they are often rather large (5 kb for XPak). But you could still include all the decompressors (XPak's is only 1 kb). One hardly ever uses two file explorers at the same time on a calculator... Why would you like libraries here ?
bremen	Re: New version 0.65 beta + XPak comparison	#20	[-]
Posts: 91 (2004-01-13 19:06:52) Reply Quote More My Recent Posts	@Pollux *Quote:* you could also add compression/decompression times (I saw a program called "ticktest" that automatically measures how long a program takes to run, to make it easier for you) That's on my to do list. However I know very well the program ticktest, it is a my program! *Quote:* you could add partial summaries so you can figure out which compression is best for which purposes This is not so simple because the files should be "mean" in a statistical sense otherwise partial results, so based on a smaller number of files, should be false. *Quote:* another idea would be to add shrnklib (I was told it performed quite well on graphics stuff), and -- just to put it all in perspective ;-) -- ziplib... Maybe if I'll have more free time: now I am officially a PhD student at the university of Genova (Italy), I am working on the simulation on a robot of the human visual attention system. Of course I accept any add to the tests I have made. I don't want to enter in the holy war TIGCC/GTC but I agree with Lionel: your compression program would be more useful if you provide at least an archive and an header files. About my compression program I think I'll soon release the source. I think would be very interesting to create an hybrid between my program and yours.

<< Previous Topic Next Topic >>

Yuku free message boards

Compression on-calc: LZFO1

Compression on-calc: LZFO1