* Initial work modifying the way named ranges are stored, and handled by the calculation engine
This should provide better support for:
- both union and intersection operators in composite named range values
- MS Excel implementation of the union operator duplicating values
- named formulae
- named ranges and formulae that reference other named ranges and formulae
- ranges and formulae that reference multiple ranges across multiple worksheets
* Initial work on handling defined names (named ranges and named formulae) correctly
- UTF-8 names (already extracted as a separate PR and merged)
- distinction between named ranges and named formulae
- correct handling of union and intersection operators in named ranges
- correct evaluation of named range operators in calculations
- calculation support for named formulae
- support for nested ranges and formulae (named ranges and formulae that reference other named ranges/formulae) in calculations
* Minor tweaks before resolving merge conflicts
* Fix extractSheetTitle() method to work on the last ! in a cell reference rather than the first
* Throw exception if a the reference to a defined name in a formula doesn't exist as a defined name
* Properly assess scope for defined names in calculation engine
* Elimination of some redundant code
* Minor tweaks to simplify entries o the stack where we need to check type
* Ensure correct scoping rules are applied when evaluating named ranges and formulae
* Adjustments to Gnumeric Reader for new defined names structure
* Initial work modifying the Ods Reader to handle named ranges, they weren't actually supported previously... this is still ongoing work
* Handle Ranges formatted as 3-d ranges, as long as the references are both to the same worksheet
* Additional testing for Named Ranges formatted as 3-d ranges, as long as the references are both to the same worksheet
* Skip composite named range tests for the moment
* Clean handling for `undefined name` exception when thrown in the calculation engine. Catch and replace with `#NAME?`
* Adjust method we use to determine whether a defined name is a range or a formula
* PHPCS Recommendations
* PHP doesn't support `mixed` yet, at least not at the minium version that we're working with
* More phpcs fixes
* More phpcs appeasements
* Final phpcs fixes for the moment
Still have a lot of echo and var_dump() statements in the code that scrutinizer will hate, but they stay for the moment while this is still WIP
* Please let this be the last of the phpcs fixes
* Unit tests to determine whether a defined name value is a range value or a formula
* phpcs appeasement
* Named tests from provider
* Initial steps for named ranges and formulae in the Ods Reader
* Reading pseudo-3d range addresses in Ods; treat second sheet reference as being identical to the first, which is the majority of cases where this will occur
* Initial work on Gnumeric reader for named ranges and formulae
* Suppress debug logging again
* Remove more debugging displays
* Last minor tweaks before phase two
* Minor refinements
* And all for the want of a space
* A little tidying up
* More tidying up
* phpcs fix
* Modify defined names in rebindParent()
* Renaming variables
* Resolve an issue with locally scoped defined names that don't contain any worksheet reference
* Keep phpcs happy
* Fix quote handling in regexp
* Fix a couple of scrutinizer issues
* Fix a couple of scrutinizer issues
* Update Xlsx Writer to work with the new defined name internal definition
Additional validation checks
* When adding new defined names through the readers, worksheet may not exist if we're only loading selected sheets rather than the full spreadsheet
* If the only thing that phpcs can pickup on is strings in double quotes instead of single quotes, then I know I'm getting close to ready
* Refactor Defined Names logic for Xlsx Writer into its own class
* phpcs keeping me on my toes
* Restore a couple of files that I managed to change without intending to
* Initial work on Ods Write to provide support for saving named ranges and formulae
* Resolve commas to semi-colons s argument separator when writing named formulae for Ods
* Extract Named Expression Writer for Ods into its own class
* Keep phpcs happy
* Refactoring of formula conversion when reading SpreadsheetML; preparation for reading named ranges because they will also need to use the same conversion method
* First pass at reading Named Ranges/Formulae from SpreadsheetML format xml files
* Remove unused namespace reference
* Defined names being written correctly for Xls; but not yet writing cell formulae that reference those defined names... that's the next big step
And I anticipate that defined names that reference other defined names will also be a problem
* Just to keep phpcs happy
... and yes, I know that there are still diagnostic echo statements in the code
* I had to miss some of the phpcs issues didn't I
* Work on the Xls Writer's Parser Tree to identify named range tokens in a formula, and to distinguish them from function tokens
* Still working on packing that d*** defined name reference in the writer
* Throw an exception in the Parser for saving Xls output if we encounter a defined name in a formula... writer will simply write the calculated cell value, and not the formula as at present
Strip out diagnostic output
* Some phpcs appeasement
* Fix a couple of Scrutinizer issues
* Additional verifications to differentiate a formula from a range value
Add explicit getters/setters for named ranges, named formulae and defined names
Additional unit tests
* Styling for closures
* Remove redundant docblocks
* Spaces
* Gah! Namespace use complaints
* Consistency of making calls to DefinedName rather than NamedRange; NamedRange should now be used only for Named Ranges, and should exclude Named Formulae
* Styling
* spurious newline
* No need to test for variable === null when we're typing it in the function argument definition
* Additional unit tests for local/global scoped named ranges and formulae; and a fix to getNamedFormula()
* Fix silly typo that led to breaking test
* Void return signature for unit tests
* Why weren't these picked up in the last pass?
* Refactoring of getNamedRange()/getNamedFormula()
* Eliminate unused constants, and defaults for private method parameters when always called with a value
* Use strict comparisons when comparing object hash codes
* Initial update to documentation for working with named formulae
* Fix for calculation of relative cell references in named ranges/formulae
* Fix current named range tests, because we should be using absolute references; tests for relative named ranges to be added later
* Fix for calculation of relative cell references in named ranges/formulae
* Updates to changelog and documentation for handling of absolute/relative references in named ranges
* Fix last remaining unit test with a named range reference
* Refactor formula conversion for Ods into a separate class; I hadn't realised that it previously wrote formulae as the MS Excel syntax without any conversion to Ods format
* Fix Ods Writer test xml to reflect Ods-native format for formula
* Docblocks
* Drop dollar prefix from Ods formulae and ranges unless it's necessary
* Set the formula convertor in the content writer constructor
* Documentation update
* Minor updates
* Remove var_dumps from file
* Fix the spurious single quote that was breaking named expressions in the Ods Writer... big sigh of relief that I finally spotted it
* Starting work on documentation for Defined Names, and some examples of using Named Ranges and Formulae
* Starting work on documentation for Defined Names, and some examples of using Named Ranges and Formulae
* Example of a relative named range for the documentation
* Mustn't have phpcs problems in sample code either
* More updates to the documentation
* That should conclude the documentation for Named Ranges, now time to move on to documenting Named Formulae
* That should conclude the documentation for Named Ranges, now time to move on to documenting Named Formulae
* PHPCS appeasement in sample code
* Initial documentation on Named Formulae
* PHPCS appeasements
* Additional comments in the documentation, and modify the named range name validation to support a \ as the first character in a name
* Fix breaking build
* Make defined names case-insensitive
* Fix case-insensitivity
* Improved documentation, and additional unit tests
* Additional unit tests, and a fix for removing a globally scoped defined name even if a worksheet is specified in the method call
* Fix unit test for removing named formulae
* Use assertCount instead of assertSame
* Forgotten voids
* Fix arguments for assertCount
* Unit tests for removing defined names, and a fix for removing locally scoped names
* Unit tests for absolute and relative named ranges in calculation engine, and fix an issue with worksheet name in the offset adjustments for relative references
* PHPCS Appeasement
* Additional unit tests, more documentation, and a fix to the calculation engine when no worksheet reference is provided with a named formula
* PHPCS appeasements
* Additional documentation and examples of using Named Formulae
* Additional examples to go with documentation
* A few minor phpcs appeasements
* Minor refactor of updateFormulaReferencesAnyWorksheet() method
* Discard an unused method argument
* Additional unit tests
* Additional unit tests
* Remove unused argument
* Stricter typing
* Fix return typehinting from remove named range/formula; should return the Spreadsheet object
* Use return typehint of self rather than explicit object type
* Redundant code just to keep scrutinizer happy
* Minor change to handle merge conflict
* phpcs fixes after merge
* Namespace usage ordering
* Please let this be the last phpcs fix needed
Co-authored-by: Adrien Crivelli <adrien.crivelli@gmail.com>
This request does not change any source code, only tests.
For a change on which I was working, a test passed when run on its own,
but failed when run as part of the full test suite. It turned out that
an existing test had changed a static value,
thousands separator in this case, and failed to restore it.
The test turned out to be AdvancedBinderTest.
The search for the offending test was more difficult than it should have
been because 26 test scripts which had nothing to do with thousands
separator nevertheless changed that value. They all changed
decimal separator, currency code, and compatibility mode as well,
again for no reason. I changed all of those to eliminate those operations.
I changed the following tests, which actually do change the static
properties identified above for a reason, to restore them as part of teardown.
- CalculationTest sets compatibilityMode and locale
- DayTest sets compatibilityMode, returnDateType, and excelCalendar
- CountTest sets compatibilityMode
- FunctionsTest sets compatibilityMode and returnDateType
- AdvancedValueBinderTest sets currencyCode, decimalSeparator, thousandsSeparator
- StringHelperTest sets currencyCode, decimalSeparator, thousandsSeparator
- NumberFormatTest sets currencyCode, decimalSeparator, thousandsSeparator
- HtmlNumberFormatTest sets currencyCode, decimalSeparator, thousandsSeparator
If row or column dimensions are accessed, then HTML writer would
still generate lots of empty cells, to show nothing at all. This
now ignore row and column dimensions to only output cell that
actually exists (even if those cells are empty).
Fixes#1235Close#1537
* Xls Writer - Correct Timestamp Bug, Improve Coverage
I believe that Xls Writer is 100% covered now.
The Xls Writer sets its timestamp incorrectly. The problem is actually
in Shared/Ole::localDateToOLE, which converts its timestamp using
gmmktime; mktime is correct. If I save a file at 3:00 p.m. in San Francisco,
this bug means the time is actually recorded as 3:00 p.m. UTC.
A consequence of this is that if you use Phpspreadsheet to read the
file and save it as a new Xls, the creation timestamp goes further
and further back in time with each generation (or further forward
if east of Greenwich). One of the tests added confirms that
the creation timestamp is consistent with the start and end times
of the test.
The major change in coverage is adding tests to save GIF and BMP
images, which aren't supported in Xls, but are converted to PNG
in the PhpSpreadsheet code.
* Fix For #1509
User expected no CSV enclosures after $writer->setEnclosure(''),
which had been changed to be consistent with $reader->setEnclosure('').
Writer will now omit enclosures after code above; no change to Reader.
Tests have been added for this condition.
* Add Option to Write CSV Enclosure Only When Required
Allowing the user to specify no enclosure when writing a CSV can lead to
a situation where PhpSpreadsheet (likewise Excel) will not read the
resulting file as intended, e.g. if any cell contains a delimiter character.
This is demonstrated in new test TestBadReread.
No existing setting will rectify this situation.
A better choice would be to add an option to write the enclosure
only when it is needed, which is what Excel does. The RFC4180 spec at
https://tools.ietf.org/html/rfc4180
states when it is needed - when the cell contains the delimiter,
or the enclosure, or a newline.
New test TestGoodReread demonstrates that the file is read as intended.
The documentation has been updated to describe the new function,
and to change the write example where the enclosure is set to null.
* Scrutinizer Suggestions
3 minor changes, all in tests.
This problem is that ZipStream, in contrast to ZipArchive,
is saving 2 files with the same path. I have opened an issue with
ZipStream, who agree that this appears to be a bug.
For the case in question, PhpSpreadsheet is attempting to save
a file with the same path twice (and unexpectedly succeeding)
because of a clone operation. This fix attempts to rectify the problem
by keeping track of all the paths being saved in the zip file,
and not attempting to save any duplicate paths.
The problem case attempted to save printersettings1.bin twice,
but there are other possible exposures, e.g. by cloning a sheet
with a drawing.The new test cases clone an existing sample which
has both printer settings and drawings.
Replace default gridlines with different style. Usable in PDF
as well as HTML.
Documentation mentioned use of setUseBOM with Html, but that method
does not exist, and there is no real reason to support it.
Removed it from documentation.
We give users the ability to edit Html/Pdf, but it's a little cumbersome
to use the edited Html for an Html file, and difficult to use it
for a Pdf. I believe we could make it fairly painless in both cases
by allowing the user to set a callback to edit the generated Html.
This can be accomplished with fewer than a dozen lines of very simple code.
I think this would be easier than grabbing the Html in pieces,
editing it, and reassembling it. I think it would also be simpler
than an alternative I considered, namely the addition of a new method
(e.g. saveEditedHtml) to each of the Html and Pdf writers.
One edit that users might like to make when editing html is to add
fallback fonts, something that is not currently available in
PhpSpreadsheet, and might be difficult to add. A natural extension to
that idea would be the use of webfonts, something which is guaranteed
difficult to add. See samples/Basic/17b_Html for an example of this.
None of the PDF writers support webfonts yet. That doesn't mean they
won't do so in future, but, for now, samples/Pdf/21a_Pdf is a prosaic
example of something you could do with this callback. In fact, this
opens the door to letting the user replace the entire body with data
of their choosing, effectively allowing PhpSpreadsheet (where you can
set things like paper size and orientation) to be used as a front-end to
the Pdf processor without the user having to be be overly familiar with
the vagaries of the PDF processor. I think this is actually a pretty
nice idea. YMMV. See samples/Basic/21b_Pdf for an example.
No code changes. The tests in all of these scripts write to at least
one temporary file, which is then read and not used again. The file
should be deleted to avoid filling up the disk system.
This test changes directory then performs an assertion.
No problem if the assertion succeeds. I was a little concerned about
what would happen if the assertion fails, leaving us in the
new directory. So I have changed test to use setUp/tearDown
to ensure that we end up where we started.
There are a number of situations where HTML write was producing
HTML which could not be validated. These include:
- inconsistent use of backslash terminating META, IMG, and COL tags
- @page style tags in body rather than header. Aside from being
non-standard, HTML Reader treats those as spreadsheet data.
- <div style="page-break-before:always" />, a construct which is
usually better handled through css anyhow.
- no alt tag for images (drawings and charts)
Other problems:
- Windows file names not handled correctly for images
- Memory drawings not handled in extendRowsForChartsAndImages
- No handling of different values for showing gridlines
for screen and print
- Mpdf and Dompdf do not require the use of inline css.
Tcpdf remains a holdout in the use of this inferior approach.
- no need to chunk base64 encoding of embedded images
- support for colors in number format was buggy (html tags
run through htmlspecialchars)
Code has been refactored when practical to reduce the number of
very large functions.
Coverage is now 100% for the entire HTML Writer module,
from 75% lines and 39% methods beforehand.
All functions dealing only with charts
are bypassed for coverage because the version of Jpgraph available in
Composer is not suitable for PHP7. The code will, nevertheless,
run successfully, but with warning messages. I have confirmed that
the code is entirely covered, without warnings, when the current
version of Jpgraph is used in lieu of the one available in Composer.
I will be glad to revisit this when the Jpgraph problem is resolved.
Directory PhpSpreadsheetTests/Writer/Html was created to house
the new tests. It seemed logical to move HtmlCommentsTest to
the new directory from PhpSpreadsheetTests/Functional.
A function to generate all the HTML is useful, especially for testing,
but also in lieu of the multiple other generate* functions. I have
added and documented generateHTMLAll.
The documentation for the generate* functions (a) produces invalid html,
(b) produces html which cannot be handled correctly by HTML reader,
and (c) even if those were correct, does not actually affect
the display of the spreadsheet. The documentation has been replaced
by a valid, and more instructive, example.
The (undocumented) useEmbeddedCss property, and the functions
to test and set it are no longer needed. Rather than breaking
existing code by deleting them, I marked the functions deprecated.
This change borrows a change to LocaleFloatsTest from
pull request 1456, submitted a little over a week before this one.
## Improve NumberFormat Support
First phase of this change included correcting NumberFormat handling
in HTML Writer. Certain complex formats could not be handled without
changes to Style/NumberFormat, and I did not wish to combine those changes.
Once the original change had been pushed, I took this part of it back up.
HTML Writer can now handle conditions in formats like:
[Blue][>=3000.5]$#,##0.00;[Red][<0]$#,##0.00;$#,##0.00
In testing, I discovered several errors and omissions
in handling of some other formats.
These are now corrected, and tests added.
There were about 20 skipped tests for RATE and PRICE marked
"This test should be fixed". This change does that by fixing
the code for those functions, validating the existing tests,
and adding new ones. XIRR and XNPV are also substantially changed.
As part of this change, the following functions also have minor changes:
- isValidFrequency
- COUPDAYBS
- COUPNUM (additional tests)
- DB
- DDB
PhpUnit reports 100% coverage for all the changed functions.
Since I was dealing with skipped tests, I also fixed
tests/PhpSpreadsheetTests/Writer/Xlsx/LocaleFloatsTest,
which was being skipped in Windows. I also delete the temporary
file which it creates.
There is now only one remaining test which is skipped -
ODS Reader is not complete enough to run some tests against it.
Unfortunately, that test is too complicated for me to deal with now.
In researching this change, I found several places in the code where special code was added for Gnumeric claiming:
- Gnumeric does not handle free-format string dates
- Gnumeric adds extra options, not available in Excel,
for the frequency parameter for functions such as YIELD
- Gnumeric rounds the results for DB and DDB to 2 decimal places
None of these claims is true, at least not on a recent version
of Gnumeric, and the code which supports these differences is removed.
There did not appear to be any tests targeted for
these supposed properties of Gnumeric.
The PRICE function needed relatively minor changes - mostly
additional tests for invalid input. The main problem with the PRICE
tests is that Excel appears to have a bug. The algorithm is published:
https://support.office.com/en-us/article/price-function-3ea9deac-8dfa-436f-a7c8-17ea02c21b0a
The results that Excel returns for basis codes 2 and 3 appear to be
incorrect in many cases. I have segregated these tests into a
new test PRICE3. The results of these tests agree with the published
algorithm, and to the results for LibreOffice and Gnumeric.
The results returned by Excel do not agree with them.
The tests which remain in the test PRICE all use basis codes other
than 2 or 3, and all agree with Excel, LibreOffice, and Gnumeric.
For the RATE function, there appears to be a problem with how the
secant method was implemented. I studied the implementation of RATE
in Python numpy, and adapted its implementation of secant method.
The results now agree with numpy, and, more important, with Excel.
XIRR, which calls XNPV, permits its dates to be earlier than the
start date, whereas XNPV does not. I dealt with this by renaming
the existing XNPV function to xnpvOrdered, adding a parameter to
indicate whether start date has to be earliest. XNPV calls the new
function with that parameter set to TRUE, and XIRR calls it with
the parameter set to FALSE. Some additional error checking was
added to xnpvOrdered, and also to XIRR. XIRR tests benefited
from increasing the value of FINANCIAL_MAX_ITERATIONS.
Finally, since this change is very test-related:
samples/Basic/13_CalculationCyclicFormulae
PhpUnit started reporting an error like "too much regression".
The test deals with an infinite cyclic formula, and allowed
the calculation engine to run for 100 cycles. The actual number of cycles
seems irrelevant for the purpose of this test. I changed it to 15,
and PhpUnit no longer complains.
I believe that both CSV Reader and Writer are 100% covered now.
There were some errors uncovered during development.
The reader specifically permits encodings other than UTF-8 to be used.
However, fgetcsv will not properly handle other encodings.
I tried replacing it with fgets/iconv/strgetcsv, but that could not
handle line breaks within a cell, even for UTF-8.
This is, I'm sure, a very rare use case.
I eventually handled it by using php://memory to hold the translated
file contents for non-UTF8. There were no tests for this situation,
and now there are (probably too many).
"Contiguous" read was not handle correctly. There is a file
in samples which uses it. It was designed to read a large sheet,
and split it into three. The first sheet was corrrect, but the
second and third were almost entirely empty. This has been corrected,
and the sample code was adapted into a formal test with assertions
to confirm that it works as designed.
I made a minor documentation change. Unlike HTML, where you never
need a BOM because you can declare the encoding in the file,
a CSV with non-ASCII characters must explicitly include a BOM
for Excel to handle it correctly. This was explained in the Reading CSV
section, but was glossed over in the Writing CSV section, which I
have updated.
* Handle Error in Formula Processing Better for Xls
When there is an error writing a formula to an Xls file,
which seems to happen when a defined name is part of a formula,
the cell is currently left blank. A better result would be to
write the calculated value of the formula.
* Making Changes Suggested in Review
Per comment from Mark Baker:
1. Made return codes from writeFormula into constants.
2. Skipped redundant call to getCellValue when possible.
3. Added support for bool type, adding additional tests
for bool and string.
Per comment from PowerKiki:
1. Used standardized convention for assigning file name in test.
Since before this change, save would throw Exception, I kept
the unlink for the file in tearDown.
* Initial unit test for locale floats
This will require potential modification of the TravisCI environment to support other locales
* var_dump to check output on TravisCI
* Fix assertions for double/float and with/without line reference
* Style in unit test
Prior to 1.10, all numeric values where read as floats. In 1.10
numeric values are read using 0 + x, which relies on PHP type
juggling rules. As a result, float(0.0) is written as string('0'),
then read back as int(0). This fix causes the writer to retain the
the decimal for float values such that a reader can differentiate
floats from ints.
Closes#1262
This will let users read a file that contains data that are not properly
supported and write them back to a new file untouched.
- load workbookProtection attributes
- save loaded pageSetup[r:id]
- save loaded sheet's AlternateContent
- save loaded unparsed VmlDrawings
- save loaded drawing files `rId`
- save loaded draw's AlternateContent
- save loaded control properties
- save loaded printer settings
- save loaded unparsed override content types (for ctrlProp, ...)
Closes#435
Because even if it doesn't make a difference in practice, it is
technically more correct to call static methods statically. It
also better advertise that those methods can be used from any context.
Mark Baker